I feel that 5 acres of garden will greatly improve you pond -- unless you are using some kind of herbicide/pesticide/...cide that is harmful to fish.
Assuming you have runoff, the unused fertilizer needed for the plants should help your pond, unless it is extremely excessive. Similarly, you should be cautious of filling the pond with runoff silt from the garden area.
Overall, I see it as a great benefit to the pond and the fish living in it.
