I will say the "20 foot from any property line for structures" is usually something you might see in a platted subdivision or a part of a OWNERS ASSOCIATION. May not have anything at all to do with your situation. I don't know the real estate law in South Carolina. It may be way different from Texas!! Did you buy Title Insurance? Is the survey a recent and certified one? Does it show metes and bounds ? Does it have the surveyors name on it?

Dear Alcohol, We had a deal where you would make me funnier, smarter, and a better dancer... I saw the video... We need to talk.