How many 4" bass did you stock originally? If you stocked the right number, the need to cull any of the original stocker bass shouldn't be there. They will for the next several years be your trophies. You will most likely never see the growth rates you will experience from the first class of fish. Seeing impressive growth rates in a new pond or one that was drained and restocked is expected, it's maintaining over several generations that becomes impressive. You could consider removing some of the bass that in that original group that are showing slower than average growth rates though. The most issues start when the original stocker bass reproduce and their offspring become too abundant and over eat their forage. When that happens usually depends on the size your bass were when stocked, how many were stocked, latitude of the pond and fertility of the pond.