Is a bowl of fruit with some milk fattening?

With soya milk no, if you don't like that then skimmed. If you don't want soya or skimmed, then yes using semi-skimmed or full-fat will be fattening. Soya or skimmed is what you will need to use if you are worried about fat.