When drilling those large holes, I think a holesaw will be your best option. Using a 2" spade bit on a curved section of 4" pipe will have such an uneven contact area that it will no doubt try and grab. If you could feed it slow and straight it would most likely work, but trying to do it by hand would be difficult.
I would probably use a spade bit below 1.25", and a hole saw for the bigger bores.
I'll bet a 2" unibit, or stepped bit, would be pretty expensive, Harbor Freight or not.
For those large holes, a drill press with lots of room around it, a piece of light duty, 4" angle iron welded to a base for use as a v-block, and a hole saw will be your best bet, in my opinion.
