When stocking just a few larger fish, are you really stocking there offspring? Will there offspring grow just like any other fish?
For the most part, this is why I thought this is not that big a deal. The 'stunted' fish will spawn, and their progeny should be somewhat normal if they have good growing conditions.
