I was told not all business is good business, and you can't be everything to everyone!
I absolutely believe that there are some jobs/customers you're better off without. Took me awhile to figure that out, what with that nasty, lurking fear of starvation and all.
But you soon learn. Wow, do you ever learn. Glad those days are behind me. Trust your first instinct....it's usually correct.