My property line goes to the center of a live creek. Years ago the owners agreed to put fences back and forth on the banks so each neighbor can have a stretch to water his cows. My neighbor told me that the previous owner to my land tried to get his little piece back but after going to a lawyer decided it cost to much and gave my neighbor the deed.
If I were you I would go to the lawyer, tell your neighbor your rights and offer to sell your neighbor the land for really cheap to not have to go to court. Explain to him that you don't want to pay taxes on his land and maybe he'll compensate you for it and not have hard feelings.