It is certainly a shame to have to worry about others feelings when it comes to conserving a resource such as a pond you have created for your own family's enjoyment. People who haven't spent the time to develop such a fishery cannot be expected to understand the labor invested and how special it is to it's owner. Real friends wouldn't take offense to being asked to limit their outings though, it just takes a little tact to deliver the news. Bite the bullet and breach the subject, good luck.