You could always invite them to chip in on the management costs. Probably that would whittle down the number of prospective fishermen significantly.

But seriously, don't hesitate to give them the boot. As CJ noted, granting all of their friends permission they don't have to grant, is the hallmark of people who don't respect your pond.