Tim,
I would leave a note, right next to the first one, asking our friendly hunter to post his schedule with you, so you can be an accomodating neighbor.
If he posts the schedule, as an arrogant idiot should, don't go along the fenceline while he's there.
Invite the game warden to ride the fence instead. Let him ride your ATV.
I know several game wardens, and I bet yours would get a kick out of "checking your fences", and checking the hunting neighbor's license, too.
Then, share the next note you get with us, after you show it to the landowner who is letting this fool hunt.