I thought I was the only one that did that!
One thing I started doing is to wear a glove on my left hand. That has cut fish drops and fin pokes down about 90%.
I first did that when I blew my left hand up with a piece of metal that flew out of a press. Once the open wound healed enough so I thought it was safe to get it wet I wanted to fish. But the last thing I wanted was fin pokes and the chance of getting infection in a hand that was still far from being completely healed.
I started off with a plain brown jersey type glove. I would dip it in clean water before handling the fish so it would be wet. It worked so well I just kept using a glove and fish all the time with one on my left hand.
I bought some regular fishing gloves, but I don't like them as good as some freebee gloves I got at a farm show. They are thin and very stretchy so they fit tight on my hand and they are made such that left or right doesn't make much difference so I can use them all for left hand. I think they may be some type of gardening glove. They remind me of those orange stretchy gloves with the rubber strips on the palm that are made to get wet. Only these are thinner with no rubber strips.
Maybe it is a bad idea for the fish slime coat, I don't know. But I have to believe it is better than dropping them in the dirt and it sure almost eliminates getting poked by the fins. I keep a gallon can of pond water that I dip my gloved hand in before handling the fish.
Here is a picture of the type glove I am using.
gloved handling fish 
