HSB are very sensitive to O2 changes, more so than the other fish you list.
I lost the vast majority of mine after a 2"+ cold rain occurred in late fall 2 years ago. It changed the pond temp rapidly and killed off a bunch of vegetation....and that killed a ton of my HSB.
I still have a few, having caught a couple this year. But it sounds like you may have lost most if not all due to some sort of winter kill.