It will just continue to build up and over time you'll lose a lot of depth. If you're doing the work, do the full job. Plus, you may be able to eliminate a lot of extra crud that is in your silt, crud that could be affecting your water, the vegetation, the fish, etc. It's basically giving you a clean slate of sorts.