My advice is go with a skidsteer. I had a small utility tractor for moving round bales and small jobs.I decided to sell it and bought a used 2001 Bobcat 773(46hp I think). That thing does everything a tractor can do and more!They even have the brushhog attachments now.Would be great for your building project. Just something to think about.
