What you describe sounds like a blue-green algae problem. Do you see what looks like green paint floating on the water? If so, blue-green algae is your culprit. Fish taste like the water they live in and the food they eat. And, musty smelling water can be a good thing. It's nature's way to remove the culprit. Send it out of the water, into the atmosphere, where nature can process it into something else.
If you don't see "green paint", look for tiny green specks suspended just below the water's surface during late afternoon hours. That, too, is either blue-green algae, or planktonic algae. Either of these situations cause musty smells and taste of fish.
Is this what you have? If so, treat with an algaecide, or leave it alone and it will pass. A "bad" case may last as long as three or four weeks. If you have a long lasting case, treatment may be the best option. Feed me facts, and we'll help.

Teach a man to grow fish...
He can teach to catch fish...