Jeff :

Stand pat for a while. In most states (and I am fairly sure Calif.) if people use property for a period of time they can claim rights in the land by adverse possession sometimes called prescription. You need to take steps to rule that out. Contact your realtor and seller and get an affidavit from them stating the facts prior to your purchase and that any use by your neighbors prior to your purchase was with the prior owners permission. If their use was with the owners permission then it can't become adverse or prescriptive. If you think their prior use may be a problem then think about that step. If not then at least talk to the realtor and prior owner and see what they say.