It might be worthwhile to explore the deed history of your (and surrounding) property. I would want to know if your property was part of a larger tract at the time the pond was built. If it was, then someone divided it along the current lines knowing the pond was there.. If not, then your neighbor may have watched bulldozers treading on his land and he's still sore about it.
Your property lines are not logically drawn. They are the way they are for some reason. Your neighbor's access to his field may have originally been an implied easement that was codified in the not-to-distant past. Country folks can be ornery until they know you are not a threat.
