I'd surely think you could start havesting middle-sized BG, too. I would want to wait until after next year's spawn before taking any LMB out, if not concentrating on large bass.
What are your management goals?
"Live like you'll die tomorrow, but manage your grass like you'll live forever." -S. M. Stirling