It depends on what temperature and humidity and what strain of Penicilium candidum (Pc) you're using. The most likely scenario that is causing the brown is when Pc begins to die it turns brown and smells more ammoniated. If you want to slow that down, you can choose a slower growing Pc (i.e.: not Pc Neige) but also adjust temperature. If you give me more info (cultures and aging conditions) I might be able to give you more specifics. But I'm also not an expert.
happy cheesemaking!