Yes, it's always about the concentration of such toxins. We eat, drink and breathe in toxins everyday. Take alcohol for example. In moderate doses, especially in red wine, it can be healthy for you. If you drink all day, everyday, expect your liver to give out at some point.

When it comes to fish, same thing. Here in NC, many lakes have advisories about eating the fish in them, specifically catfish. The government recommends not eating more than 1 meal per week of wild catfish, and that pregnant women and children shouldn't eat any of it. Like alcohol, I'm sure if you ate catfish 3 times a day, 7 days a week, you might get sick from it.

We shouldn't stop breathing just because there's CO2 in the air.