I'm sure this has been said, apologize as I haven't read all replies, but I can empathize with your situation. I am very empathetic as well, not wanting to hurt someone's feelings or really cause anyone any discomfort if possible. However, if your friends do not respect your wishes, then I think you find out a great deal about them. If they are reasonable people, they will understand. If not, well you know what you have there. If I had access to a honey hole and the friend who granted me access shut me out, yeah, I'd be disappointed, but it's a privilege, not a right.
Might I suggest something as an alternative to all or nothing? Maybe have a get-together with those friends you know who fish your pond once in a while, say a few times a year, and invite them. That way you're not completely shutting them out, but you're also still very much involved in controlling their effect.
