I am a geologist, NOT a dam expert, or a civil engineer. However, if that soil was compacted during construction, then it sure does look like you are suffering soil losses due to internal piping.
I think the most common internal piping route is along the outlet pipe - which is why seep collars are installed. You said this is on the opposite side of the dam, so that is not the problem.
Noel's link shows some illustrations of the other types of piping scenarios. Did you have a bed of Indiana limestone underlying your topsoil layers? If so, you may have a fracture open in the limestone. When the soil is saturated, soil and water both migrate down into the fracture. The resulting void makes it easier for additional soil to find its way into the fracture the next time. Hence the "piping" description.
Another possibility is that you are connected to an old drain tile system from when the land was farmed. Did your acreage used to be farm land? If you don't know the specific history of your ground, is there a farm neighbor you can ask to see if he has drain tiles on his land?
You should have cut some of these tiles or pipes during construction if they were present. Maybe ask your contractor if he saw any?
Another possibility is that you have piping occurring into the pond itself. Soil and water is flowing from your bank into a lower level in the pond. If you are able to easily drain the pond, you should see some weird feature or even a mud volcano on the bottom of the pond.
Good luck, Rod.