Many competitors avoid eating dairy or fruit during contest prep because of a few unfounded fears, mainly that they prevent you from getting super lean. Fortunately, this is a myth.
When it comes to fat loss, calories are king. Although dairy can increase insulin production (just as protein can), changes in insulin levels don’t supersede the importance of calories for weight loss. If they did, low carb diets would be superior to calorie-matched high carb diets—and they aren’t. Similarly, the sugar in fruit will not impede weight loss.
With flexible dieting, you are free to choose the foods you like and enjoy. The most important thing is that you hit your target macros.
While you may be wise to avoid certain foods if they cause you GI distress and bloating (just as someone with a food intolerance would do), the theory that certain foods will prevent you from getting lean is not evidenced based and should be disregarded.