I've read that touching fish with your hands can cause them damage to their slime layer exposing them to greater risk of infection, parasites, etc.