Says the water must be coming up from the ground because of my pond. How could he prove that? Just that when my pond is full his lawn is swampy. Low pond level his lawn is dry.
Let me guess, when it rains a lot - your pond is full AND his yard is swampy. When it doesn't rain for a while, your pond is low and his yard is dry.
That is exactly what would be expected if your pond was NOT the source of his yard water. Proving the connection would be difficult.
However, he doesn't need to prove it to sue you. All he needs is suspicion and a filing fee. You would then need to hire a lawyer to respond. (I am not a lawyer, but my wife is. Do not take legal advice on the internet, only take it from a lawyer that you have hired.)
That being said, I believe you generally can be sued when you divert water OFF of your property and onto a neighbor's property. An example would be when you build a berm or levee in your yard to keep water that previously ran down a slope diverted away from your basement - and the water that should have run across your property now runs across a neighbor's property.
Are you using a well to pump water to your pond? If not, and your pond actually captures surface water runoff to gain water volume, then you would NOT be diverting water to his yard.
It is theoretically possible that your pond could be the source of his yard water, but it would require nearly perfect geologic circumstances.
If you reach the point where you think he is about to bring a lawsuit, then I would start some work on proving your case. After he files, you should have your lawyer make an offer to have a witnessed test of pumping down your pond - where you pump down your pond a few feet, and his yard stays wet.
Of course, a good lawyer never wants to ask a question for which they don't already know the answer. Therefore, I would advise you run your own test ahead of time (if the neighbor can't observe). Some time when his yard is soggy AND you have solid rains in the forecast, pump down your pond a few feet and see if his yard remains soggy, and for how many days. Hopefully the rains will then fill you back up.
If you get very positive results from your "test", and your neighbor is still angry, then perhaps offer to demonstrate a test to him before he files a lawsuit. It is no fun to have angry neighbors if you can possibly "smooth the waters".
There are a ton of assumptions in my post above. If it does not apply in your exact situation, then disregard it as bad advice!