Welcome Carlyle!
Your question has been tossed around this board in the past and all previous posts offer good advice but I'm going to saddle up with salth2ocowboy. My ponds are open to my neighbors, but the offer comes with rules, verbal not posted. Violate the rules and the priviledge will be taken away. It gives me great satisfaction to see my neighbor(s) and their son/daughter drowning worms in my ponds. You should be thankful(my opinion)that your neighbors asked for permission as opposed to exerting their "right" to fish in your pond. Perhaps some stewardships roles could be developed to the benefit of all involved.
I recall an article a number of years back that addressed this same situation with a unique twist. The pond owners did allow neighbors to fish their pond but only at certain times. When the invitation was open, they would hoist a green flag up a flag pole. When the invitation was closed, they hoisted a red flag. Feeling neighborly, go green. Seeking solitude, red is the color of the day.
My BIGGEST concern/fear with an open invitation is of course liability. This alone may swing your decision but whatever you decide, good luck with your pond and hopefully you'll become a frequent contributor here.
