I will not be as politically correct. Your neighbor is being an idiot he should be thanking you Rick. You are adding body mass and probably some antler inches and keepign more deer in the area due to your efforts. Corn is not ideal but dang awesome in the cooler months for the carbs. This means deer are in much better body condition. The deer instincevley will eat some browse for digestion. The privet back in your woods is just fine. Ask the guy what deer do in farm country with acres and acres of corn? Geexz the nerve of this guy.
Also I do not buy into the CWD scare. They prdicted it might kill all the deer even where it is detected less kills than blue tongue that took its tole on my KY property.
If you want to put some more inches of antler for him then fed deer food otherwsie do what your doing so you can enjoy watching em.