Be careful about gloating or being too happy. I know you aren't, but don't be too confident. There are many many ways around zoning, as well as many things which may be legal but even less desirable than the camp site, in case the neighbor gets ugly. I do hope you all offered to buy him out. Of course, you could then set deeded restrictions and resell to someone who better suits your taste. The owner could very well give up, depending on his age and personality, but it's 50/50 that something else is coming, and that he will do something which could be as undesirable or more. It may be swine/hogs/poultry. It could be a make shift dog kennel which brushes on being illegal but never provably so and always loud. It may be a legal 24 hour gun/shooting range. It may be motorcross. It may be a subdivision of manufactured homes. I don't know your area or zoning regulations. I would certainly try to buy the land, if there is any way possible, and if it is a financial burden, restrict it and sell it.
Good luck.
