Well you are absolutely right, there might be undesirable molds. But yeasts/contamination is not the same as molds.
Think about it like this... your goal is to take milk and remove all the water necessary to form a condensed version of the milk, which due to the protein and fat, takes the form of cheese. If you have airborne anything in the air, such as yeast or mold spores, they will land in the milk and that may ruin a cheese. However, after a cheese is formed into a mass, the only thing to concern yourself with is the surface contamination. Here, your level of care matters. If you form the cheese and it has cracks going inside because you did not press cloth well enough or hard enough to form an even surface, the mold can ruin the inside paste. If the surface is conditioned properly, then the mold stays on the outside and its enzymes break up the cheese paste.
There are many molds that will create off flavors. Those are no good. And the difficulty is knowing what you have naturally. That's why many will inoculate with commercial strains to begin with. And another difficulty is that some molds work fine on low0moisture cheese because the proteolysis rate is slow, and work poorly on high moisture cheese.