In that design, the outlet pipes aren't what set the water level. If you were to get a 100-yr rain event, the pipe blockage could become a crisis. It has obviously managed ok with the metered flow over your weir not significantly exceeding the reduced flow capacity of the pipes. Why risk it further? Clean it up and rest easy when it rains.