Channel cats usually have to be pretty large to start spawning, so I doubt your fingerlings could have even had a late spawn last fall.
One option is that the small cats are actually from your stocking of fingerlings. When fish are introduced to a new environment, some percentage may actually fail to "thrive". It may be that your small fingerlings are surviving, but are not effectively feeding to put on weight.
Another option is that they are just the males from your fingerling stocking. Male CC are typically smaller than females. Your 4" fingerlings might have included 5" females and 3" males. The males may still be a little small, but hopefully will start putting on weight this spring!
Finally, you may actually have some bullheads in your pond. Is it a new-built pond with no connection to creeks or other ponds? If there are connections, then you could have bullheads from those sources, or there is even a slight chance that your fish supplier had a few bullheads in your order. (I doubt that latter theory, most suppliers do a great job raising their channel cats.)
I see "little catfish" in our creek every spring. I have both channel cats and yellow bullheads. They are very easy to distinguish when you can observe them swimming from a side viewing angle. The CC have a very deep "V" in their tail, even when they are small. Yellow bullheads have a perfectly rounded tail. However, you could have black or brown bullheads - they have a very slight notch in their tail, but certainly much less than a CC.
Hope that helps a little. Glad to hear that a large number of your CC fingerlings are growing rapidly. They are an amazing fish in small ponds, with an impressive growth rate - especially if you give them supplemental feed! Have fun catching them in 2026!