If anything, maybe remove 3 or 4, but really, they are not a bad fish to have around. If you do feed, you will lessen their denuding of the pond.
One other thing I would mention is that the infestation of weeds each year can be by different weeds from year to year. In my experience with only one pond over 11 years, the weed species are not always the same.