If they're small, I flip with two fingers (but like Pete said, you should be patting down the velvet anyway, so finger prints don't matter). For larger wheels, I place a draining mat on top and flip over right onto that, so you never even touch the surface directly.
I have found that if the drainage holes are small enough, you really don't need to pat down the velvet on the larger surfaces, rather just the sides.