Dang, it sounds as though if you don't actually live on site at the pond that there are some real problems when you are not there.
Personally I think it's a good thing to let people fish and selectively harvest some fish, but you have to be carefull, give most people an inch and they will take a mile. Keep the list of people you will let fish short and if they violate your trust then don't let them come back again, friends and neighbors included.
