1st - At least they ask, which is good.
2nd - I wouldn�t feel bad saying no. I doubt most folks really expect you to say yes. Myself, I would just be pleasant, but firm and have some succinct explanation ready if needed. I imagine word will get around and requests will taper off. Harder for me is letting a friend in and then they invite others next time. Others then think they have permission and things quickly get out of control.
In the end.... your puddle, your rules.