I sympathize with you, TJ!
I can't count how many times I've cleaned out a bluebird house where the sparrows had nested right on top of the natives and kicked them out. They do the same to our purple martin house. They also infested our barn until we taught a cat to climb a ladder to get to their nests.
I don't have a good suggestion for eradication. There are certain seeds you can use that they don't like so much. But if you're an avid birder then I assume you already know about those.
Squirrels have adapted to love black, yellow and white millet, and even stale bread. Trying catfood for a while, but imagine the jays will hammer it.