I think jludwig has a great idea that could make it much easier to reach an agreement!

Deepen the neighbor's pond a few feet and use material to make a berm that is 10' wide on each side of the property line. (So both of you can mow, do landscaping, etc. on "your" side.) Still get an agreement in writing.

If it works out, you could set his pond water level a few feet lower than yours. A little seepage or a 2" weep pipe could help keep his pond "full" in the summer and then he could have a tiny pond, rather that a large mud hole.

You then have full control of your OWN POND! Good luck.