EastTexasWoods,
Don't panic yet! The gravel is most likely an ancient creek or river bed that meandered away or dried up. Is there a fair sized stream or river near by?
Sand is usually the real villain. Most aquifers follow sand or sand/gravel veins of substantial thickness. A thick sand vein will suck your pond dry, and the pond's water level will rise and fall with the water table. Gravel... especially mixed with clay, is usually not indicitive of an aquifer, and not a problem. You didn't mention water seeping into the pond, so that's a very good sign.
I'm guessing your pond is just a hole in the ground, because there aren't many terrain features in your parts. Your options are limited. Your pond must be dug deep enough to stay full and cool, so you will probably have to go for it. Good luck!
