If there are sections of the dam with sufficient patches of rocks such that they are in point-to-point contact, then that is a leak pathway that is VERY difficult to seal.

I have had one "crazy" idea for a situation on my property similar to what you are describing. Hopefully, the experts on the forum can evaluate its chances of success.

Once you have excavated your long trench, you will have an exposure in your dam that is exactly the same shape as the way vendors sell pond liners! A piece of liner that is 10' wide and 80' long is not that expensive. I would consider hanging a pond liner on the downstream wall of your excavation. Bring the top edge of the liner up above the normal water elevation and then pin it in place.

I would then find the best clay anywhere on the property, or even consider trucking in some clay. You wouldn't need that much clay. You just need a u-shaped sealed area of clay to seal the edges of your pond liner. Pack two feet of clay along the bottom edge. Also seal the two vertical edges with good clay as you backfill your trench.

The nature of the rest of the backfill material should not matter much, since the pond liner will be doing most of the sealing work. (Do make sure you exclude the sharp-edged rocks so you do not puncture your liner.)

I have no clue if this will work. It is just an idea in my head, NOT a proven pond practice.

Good luck getting your pond sealed.