In My Humble Opinion = What I think, but don't sue me if I'm wrong; your mileage may vary.
100 stocked so far = 400 /acre.
Total stocking for fingerling BG usually ranges from 500/acre (Northern rate) to 1000/acre (Southern rate, especially if concentrating on LMB growth). So you have not exceeded commonly recommended levels of initial BG yet.
While your current 1"-3" BG grow, I'd concentrate on catching and sorting the larger BG available from your neighbor's pond. I think I'd add 30 to 50 of those.
If in your situation, my LMB stocking plan timing would depend on when you get a BG spawn. If the transferred BG don't spawn this year, any LMB you transfer will be trying to live on the smallest BG you are putting in - rough on both you and the bass.