Are the "floods" typically only 1-3' above the elevation of your pond?

If so, you could build a berm around your pond. A 3' tall berm would not be that expensive, and would not need to be sealed. (It would be acceptable if it leaked a little water during a flood stage, just not silt and invasive fish.)

If it then took a "five-year" flood event to reach your pond, your fish management and swimming investments would probably be cost effective over the long term.

Do you look over the pond from your house? If so, then a berm would probably significantly devalue that view.

Finally, in many jurisdictions, you are not allowed to build a berm (raise any type of embankment) in a floodplain. If you do plan to move dirt, then make sure you are within the regulations for any changes on your property.