Sorry to hear about the predicament... The police should be more receptive than telling you to put up a fence. However, it's not always the police departments fault. The amount of hoops we have to jump through to prove trespassing is insane. The police may just be completely making sure your neighbor is aware he is trespassing so that when he does trespass, your case against him is airtight. Judges and juries are often sympathetic towards trespassers for a myriad of reasons. Work with the police department to find out exactly what they feel will be the best methods to assure a successful prosecution. Game cameras are good, but unless the subject is close by positive identification can be difficult. Remember, when in criminal court the standard is beyond reasonable doubt. Sometimes that is a very tough standard to hold up to. A high quality video camera system set up with a motion detector may seal the deal. I worked with a land owner who had ATV riders tearing his land up. Through the proper posting of his property and the use of game cameras and video cameras we were able to successfully prosecute 7 different individuals. Including one who was charged 4 times with trespassing and convicted each time! Finally 6 months in jail and he decided to stop. Some people are insane and no matter what you do, they think they have a right to use your land. Sometimes sitting in jail for an extended period of time is what it takes to send the message home.