Just thinking out loud here, could the larger SMB be gorging themselves on YOY SMB and puking them up? When I pen raised my SMB I BG trained them, after feeding them 7-9 dozen small BG there would be a dozen or so BG that were regurgitated and floating in the pen the next day.
Maybe once the water temps got warmer in the spring the small number of big SMB are gorging on the smaller SMB? If the larger SMB are still there then the numerous small SMB would make them difficult to catch. Lot's of food, little competition. How big are the smaller numerous SMB, 4", 6", 8"? What sizes are you seeing when a fish kill occurs?
The other possibility is with too many small fish there simply may not be enough food to keep them healthy, if some sort of stressful event came along they would likely be the first to succumb to the stress.
