I agree with Sunil, there are a lot of Pond Boss members that mention swimming in their ponds and probably more in total based on the backgrounds of pictures where I see swimming docks, ropes with floats, etc.

I suspect a family swimming outing will spook the trout (but probably less so as they get used to the activity). Perhaps make a rule that the trout fishing people get first crack at the pond, and then the swimmers and aquatic adventurers can go after?

Good luck on setting up your new ponds for maximum family enjoyment!