When my cow was early in her first lactation, I wasn't great at getting all of the milk out of her, so sometimes it took a few days for me to have enough to make a batch of cheese. (I always make 5 gallon batches) One time, I was making citric acid mozzerella and the oldest milk in the batch was a full 7 days old. It did not set up well and it was grainy, as I remember.
A fellow member of another forum mentioned having the same trouble and it turned out that we both used some "old" milk. For your next batch, try using fresher milk and see how it turns out. I think that the older raw milk is, the more acidic it becomes, which can give some trouble.
Don't get discouraged!