I think trees producing growth inhibitors is fairly common. I know for sure that some oak trees exhibit this trait. My wife got upset when I removed all the small oak trees from around our new pond, but I didn't want to take a chance on acids, toxins, and sludge getting in.
OTOH, if your pond is large, I wouldn't think 3 or 4 trees would produce enough leaves and husks to be toxic. If the pond is small-ish, it's bound to have at least some effect. Collect all you can, and don't worry about the rest if there are no signs of stress.
