Even with as few as a couple of dozen original adult brood fish you would have "aa" matches at a low enough rate to be not noticeable. Random mutations are also occurring through time. This is probably where many of the undesirable traits came from in the first place.
If I'm not mistaken there are a few notable mammalian species that are pretty high-strung and have problems with disease resistance, etc. The speculation is that at one time in history the cheetah population may have been down to a critical few individuals, and consequently have really poor genetic diversity.
