I'll take a novice's stab at it based on the limited information. Age, construction and stocking history might shed more light on the cause of your problem.
I assume the catfish are blues? Any idea how many 5+ lbs? They're so voracious that most pondmiesters here won't stock them if the goal isn't big catfish.
My guess would be that your catfish are eating everything that gets some size to it. Going by a relative weight chart, 20lbs puts them at about 36 inches. If we assume that their feeding habits are similar to bass, then they'll target prey 1/4 to 1/2 their length. Taking the assumptions further, the blues will prefer to target the slightly more fusiform shape of the bass and crappie as compared to the bream. Your one saving grace is that you've got enough small predators to prevent catfish recruitment on a large scale and the ones that make it will be vulnerable to cannibalism for a long time.
If I had 20 acres that wasn't meeting my goals, I'd probably get it shocked and have a professional plan done up. Intense harvesting of the big catfish will make a difference but maybe not to the extent that you want. Forcing your BOW to be optimized for big crappie rather than big catfish isn't going to be quick, easy or cheap.
