Chad, it sounds like you've got a somewhat bass-heavy pond in which, as you surmised, some of the bass are still doing okay on a combination of the bluegill and the small bass. In ponds in which bluegill growth is maximized, which sounds like your pond, you won't see many 3-6" bluegill because most get munched, and the ones that don't get munched grow fast enough that they stay in that size class a fraction of the time they would in a pond in which they're more crowded. Also, any pond that is even somewhat bass-crowded, which it sounds like yours definitely is, is not going to have as many bluegill present, again simply because most get eaten by the bass. You can increase the numbers of bluegill without hurting their average size by fertilizing the pond regularly from March through October, usually once a month other than in March when you should do two applications (typically) due to the colder water. Fertilizing will also help the bass by amping the entire food chain, including more bluegill YOY produced by the mature bluegill due to a better food supply. Or, you could install two or three or four automatic feeders, which would have the same effect: more good-sized bluegill but also more forage for the bass in the form of more intermediate bluegill. The larger bluegill already present in the pond would get huge, and more bluegill would make it past the size range that is currently getting mowed down by the bass.

You could fertilize and feed both, but you would want to be careful not to fertilize except when visibility exceeds 18" so you don't produce too heavy a bloom, which could cause a fish kill.