The only issue with fridge brewing is that theroticaly you will start to breed out the cultures that like a slightly warmer environment.
If really like the idea of fridge brewing , I would brew a cup or 2 on the counter over night each time you do the 1 gallon batch then after 24 hrs add that to the mix in the fridge , this should help give the warm loving guys a chance .
My wife did the exact opposite thing by accident , she was brewing on a seed mat at 85 to 90 degrees and bred out some of the cultures that like room temps , then the grains just kept shrinking each week . She ended up having to get new ones.