Every day without enough calories is a day of growth lost. Can they grow large? Maybe but don't worry about it affecting their genetics if it's only a generation or so.
You bring up an interesting point: While the genes wouldn't change, perhaps the expression of those genes would, usually described as epigenetics. Fish stunting due to inadequate food might lead to adaptations in progeny designed to minimize growth so as to maximize scarce calories for survival. Not saying that's the case, but it might be. Would be an interesting research project.
