I have done some work around Memphis and soils in some areas can be suspect. My suggestion is to read the soils as you remove them and make sure the pond bottom wasn't sealed with a clay liner. If all the soils are native soils, and they are clay-based, you are okay. If not, you may move away the best soils only to find dirt that someone else covered up.

Teach a man to grow fish...
He can teach to catch fish...