Peter Quill actually only had trouble holding the infinity stone because he’s half Terran.
A fully human Terran would have done much better.
I mean, Jane Foster had the aether inside of her in Thor 2, and that seemed pretty potent (and potentially could have been another infinity stone, besides).
In fact, considering all of the mutants and badassery and whatnot that abounds in the Marvel universe, and the heavy-hitters Terrans can produce with just a little genetic tweaking or gamma radiation, it’s possible that Asgard took to ‘safeguarding’ Midgard in the first place because Odin figured out that the natives are scary as balls.
Maybe during the war with the frost giants, certain humans started manifesting strange/special abilities to deal with the threat, and Asgard’s scientists/wizards/etc figured out that the poor puny Midgardians were like a sleeping monster, and the aggression of the frost giants was 'waking them up’.
So Odin stepped in to stop them and to try and keep people from interfering with Midgard as much as possible, so the monster would stay sleeping.
Which is actually part of why he is so very pissed off at Loki for the events of Avengers, and at Thor for bringing Jane to Asgard - it’s a very, very tight secret that Midgard is a powder keg full of potential super-warriors, and Odin does not want that getting out, and he does not want them getting out. If Earth can stay as a nice, quiet, backwater planet with very little contact with the larger universe, that’s safer for everyone involved.