I take it just powdering a brick wouldn't be too hot of an idea?
I'll take your question as a serious joke!
The potential problem with a powdered brick is that you don't know what all is in it. From a functionality standpoint you could beat the crap out of the brick with a sludge hammer and run it through a very fine mesh sieve and come up with an appropriate powder. My concern with heavy metals is that you are coating an acidic material which dissolve metals. Then osmosis might cause migration of the metal ions into the cheese. Is this a strong possibility? Probably not but since there are other, safer materials why take the risk?
One clay you might look at is called Red Art Clay. It is sold in small bags by most pottery shops. Here is a link to a Red Art Clay MSDS:
http://www.standardceramic.com/MSDSmaterials/Redart%20Clay.pdf It contains crystalline silica as a hazardous material. This is chemically sand but in a form where repeatedly breathing it could cause silicosis a sometimes fatal lung disease. Once mixed with oil the hazard is eliminated and eating a little silica is harmless.