When you're judging, you really don't have time to think about leo colors or whether you like the coach, since you're so busy keeping up with shorthand (and most judges have shorthand for specific deductions or problems in the skill/routine as well as the elements) and deductions. During the 3-day meets in particular (or long 2-day meets), you don't care what gym is going, as long as there is always someone on the equipment!
Judges can judge their home team, but they cannot be the head judge. I've judged my gym quite a few times, and it has always been fine. After one of the home meets, I was told that the parents were talking about my panel and how consistent we were throughout the day. Judging is a lot like gymnastics in that you get into a zone.