Even if the channel cats do muddy your pond up, does it matter?
You will be harvesting them all anyway and when you do the water will settle again.
I say go for it. I have seen them raised in 50 gallon drums so I don't think that your pond is too small.