Sounds like he might be saying he could take out the part of the dam on his side if you don't please him. Having the pond that close to the property line would make me a little anxious. But, if he does something on his property that damages your property, he would be liable. Kind of like cutting down a tree in his yard that falls into yours, damaging your property.
