I urge caution. If you don't get a bloom you aren't going to kill your fish. If you overfertilize, a fish kill is a real possibility.
It is unclear from your post how much fertilizer you've added, but the way I read it you put in 40 lbs, then 20 lbs more. That should be PLENTY to get a bloom started, probably twice what you should have used. If your water temps aren't consistently over 70 degrees, and averaging in the upper 70's your bloom may not have happened yet.
How did you apply the fertilizer? My experience is that if it's not broken up and spread lightly, it can clump and rapidly sink to the bottom. I've been having good luck dumping it into the propwash or sprinkling it off of docks.
Last thing . . . have you had any big rains that might have washed out your fertilizer?
