The same thing happened to me for the first 20 cheeses or so that I made. I have no idea if any of this applies to you, but I finally figured out (after a 5-month hiatus) that I was doing a number of bad things.
1) I was WAY too cavalier about temps and cooking times - so I was more or less overcooking the curds on most of the ones I made. Fixed that by erring, if needed, on the side of lower temp and less cooking if they "seemed" done.
2) I was really beating up the curds at the beginning of cooking, so I was losing (I think) all sorts of butterfat. Fixed that by, ummmm, stirring with my hand veeeeeeeeeerry gently for at least the beginning of the cooking process - but usually all the way, since hey, I've already stuck my hand in, so I'd rather feel the curds anyway. And it's easier to separate any mats with my fingers than with a ladle.
3) I also started doing a pre-cut on the curds (I think this has to do with butterfat loss too?). So if I wanted to end up with 1/2-inch curds, I first cut (vertically) to 1", let them rest 10 minutes, then do the 1/2-inch cut vertically and horizontally. I've also starting erring, if necessary, on the side of bigger curd cuts rather than smaller - less chance of me mucking up the moisture loss that way.
Either way, whatever the technical reasons for my boo-boos, all this in combination has solved that problem for me. I hope this helps, because oh, I remember that feeling every time I excitedly broke into a cheese to discover crumbles.
