It's not a concern for coldwater species from my experience.
You swam to reloate your diffusers? I tie a polypropolene (sp?) rope (yellow in color easily available) to my diffusers and leave them on. When I need to move a diffuser I pull the ropes.
The only time this guy gets into cold water is when he falls through the ice!
