One thing to do is simply to say nobody fishes the pond unless you are there, and no "friends of friends" allowed.
Depending on the size of your pond, excess fishing can simply kill the experience for you & your family.
Just tell the truth; no need to beat around the bush.
One member who has been away from the forum lately, saltH2OCowboy, supposedly let his neighbor fish his pond. One day his wife saw that some other guys fishing who claimed the neighbor said that Salt didn't care if people fished his pond. They were filling coolers with fish that he paid for. There might be some details that I left out, but you get the picture. Salt travels a lot so he's away from his pond a lot.