The truth works the best!
Tell them what you've said here, that you don't mind occasional outings but they are rare. Add that you are trying to manage the pond and it would be nice if they helped with a creel survey sometime.
What is selfish about spending your money on your land to raise your fish? If your buddys think fishing is a requirement, fugettaboutem!
Absolutely, Tell the truth and express your concerns. They will not be friends if this upset them. They will be friends if they respect you and your decision. Thats how you thin them out.