Dirt is full of nutrients. If wanting a swimming only pond, consult a swimming pool supplier and treat the pond as a pool...remove the dirt also, if possible, and replace with cleaned river rock or sand if needed.