All of the above answers are very good. I'm new to making cheese too but I have been gathering supplies for a few months now. The experienced folks here encourage new comers like us to find a cheese (or cheese family) and get to know it and work on perfecting your cheese. Not only does this allow you to become a better cheese maker but it limits all the have to haves. I'm such a noob, I throw that good advice away because I want to do it all and make all kinds of cheeses. As a result I have lots of failures and spend a lot of money on this than and another thing. Perhaps I'll settle down and take their very good advice and learn to really be better at one kind of cheese but for now I'm having fun and learning lots by comparing how little differences make such different kinds of cheeses, and eating a wide variety of cheese when I have successes. (I should add that I have a Guernsey cow and a lot of milk to play with)
I keep all my supplies in a rubber made like box to keep clean. When I look at what I use the most it would be my4 gal pot, the curd cutter, a stainless cheese ladle, a stainless long handled spoon, a stainless colander, a good digital thermometer with a probe and a temp alarm (what an improvement from a hand held!),cheese mats, aging boxes, good cheese cloth (but am switching to plyban). I also get a lot of use from my home made cheese press. I am
so glad I didn't buy an expensive one. Because I like to play with a variety of cheeses I keep a variety of cultures (many use buttermilk and yogurt and are happy however), a hygrometer (cheap from a cigar supply place) and liquid rennet. I have a variety of molds, such as Camembert, that I use sometimes but the one I use the most often is the food safe water pipe that we drilled holes into for my home made press.
What I will buy next is a larger pot (8 or so gal) because larger cheeses age better. I will also get another refrigerator with an external temp controller to make an aging cave because a large part of my cheese failures have to do with it is the aging temps. I've tried to skip on the cave but in the summer I couldn't make aged or ripened cheese without it. I'm also working on getting a vacuum sealer because the wax just isn't working well for me. The list goes on from here but these are the most pertinent for me.
These are just my suggestions and again I'm not the most successful cheese maker around here
but let my failures be for learning.