RobA,
The quick easy answer is yes, you could be sued if someone was injured. There have been law suits filed against property owners by people that have been injured on their property while tresspassing (obviously)without the permission of the owner. Our common sense and values tell us this is not right and should not happen but in fact is does happen. If someone is injured on your property with or without your permission you could still be sued, now wether they can win the suit is another question, but you could be out a lot of money just defending a suit. Most liability policies will pay to defend you in addition to paying for a settlement up to your policy limits.
I have mentioned this before on the forum, but you want to "BE SURE" that your underlying liability policy and Umbrella Policy will provide coverage for "Farming & Ranching" exposures. Please check with your agent and get it in writing because I know for a fact that some Insurance Companies policies exclude coverage for these operations. You do not want to find this out after the fact.
Most churches have a liability release or waiver they have people sign before attending an activity. If it were me I would have the church include me (or my entity) on the waiver. I would also ask the church to provide me with a Certificate of Insurance from their Insurance Agent showing my as an "Additional Insured" under the churches liability policy. There should be no additional charge to the church to have this done and it would make their policy primary.
It is a shame that we have to consider these things just to share our property with a church or anyone else for that matter.