I get raw milk from a farmer who is not overly concerned about cleanliness. In his defense, he just throws the milk out unless I go to get it. That said, he is willing to indulge me and my OCD with cleanliness when I go to get the milk. His cows are healthy. So, true, I am pretty careful. But, other than a few specks in the filter, there are never 'floaties' in the milk. How would it get in there.. unless he is hand milking into an open bucket? Just a speck of debris can contain millions of bacteria. I agree. Would find a new source. Once in a while I have a breach. ie, the local dog dropped his toy (that he wanted me to throw) right in my milk jug. I tossed that milk.
Or maybe there is the possibility the guy would work with you. You could see where the problem is and make suggestions to correct the problem. Of course, this only works if you have a good relationship with the farmer. Good luck!