Buzz, there is no point in trying to get along with this neighbor, and he'll continue to cause both me and the pond owner trouble until he gets what he wants. I've seen this happen before to others and there is no pleasing these people.
A friend of mine and his kids had hunted a farm for well over 20 years with no problem. One day a new neighbor moved in next door. He also secured permission to hunt the farmer's land. He began telling the farmer stories about them shooting near his house, parking on his property, seeing them "sneak out" "poached" deer, and even went so far as to call the sheriff to report their car as being "abandoned". They knocked on the guy's door one day to try to straighten out the problem, and he responsed by telling them he wouldn't be happy until they stopped coming out to hunt "his" woods. Words to the effect of him being local and them being outsiders. The farmer knew what was going on but would rather please his neighbor than allow this family tradition to continue. I figure I'm saving myself and the pond owner a lot of grief by ending this early. I'm not happy about it, but I figure when this guy's controlling ways starts to make enemies out of them then I'll consider returning.