We dig a portion of our pond bottom each year and use it to make raised gardening beds. Our soil is really just a clay that grows very little vegetation as there is next to no organic matter in it naturally. The parent material is Mancos Shale. We live in a "pigmy forest" of pinyon pines and juniper trees. Rainfall is scarce; grass and understory shrubs are rare. It's mostly bare ground with the trees as over story. Soil pH is high-greater than 8.
What has worked for us is to mix pond muck at about 50% with wood shaving bedding from the chicken house, wood chips (I have a big wood chipper), and the natural dirt from the site.
With no amendments to the soil, corn grows 8" tall! With manure and compost, corn will grow 2' tall. With the mix of pond muck as listed above, the sweet corn thrives, tomatoes are loaded with fruit, cucumbers go wild.
There is some sulfur smell to the muck, so it's fair to assume the sulfur in the muck is lowering the pH some to a more plant friendly level.
I think your spoils could be valuable to the right people-gardeners, landscapers, orchardists.
Roger