Yeah, the foxes are much larger than the greys. And they can be tough to skin, especially if they're old enough to have come over on the ark with Noah. I try to pass on shooting those!
I find greys to be a more challenging target than foxes. They're smaller, they move around more, and they spook easier. You startle a fox and he'll quiet down for a few, then likely start back up, giving you another chance. Startle a grey, and you may not see or hear him any more that day.
Definitely keep your squirrels warm, and on the smaller side when it comes to skinning.