It could be a number of things - not enough salt, too much acidity, bacterial contamination in the milk, etc. And some cheeses can go through a bitter phase but are fine after additional aging (say, 5 months total) but Colby should be good at that age.
Which recipe did you use?