We did. I don't think it does your dam any good wrt compaction and sealing, and it's too valuable to let disappear. We made four topsoil piles; one got spread over the completed dam, one went for improving upstream landscaping, one I am pulling topsoil off as needed for whatever, and one is in reserve.

"Live like you'll die tomorrow, but manage your grass like you'll live forever."
-S. M. Stirling
[Linked Image from i.pinimg.com]