I would wager it's not the feed itself although it could be. If that was the case then fish farms that raise fish for consumption would not exist. That said, there is something in the fish farming industry known as "off flavor." It can be caused by the pond's chemistry itself or blue/green algae or other factors. Fish farmers typically "purge" their fish before they go off to market especially if they do a taste test and find there is an off flavor. They simply put the fish in tanks with clean well water and don't feed them for a few days. That usually gets rid of the off flavor.

If pigs could fly bacon would be harder to come by and there would be a lot of damaged trees.