Look, if I spent 14 years of my life completely devoted to, lets say, teak-wan-do, spending literally every waking minute improving my skill, who are you going to bet on if I was assaulted by 4-6 street muggers (with no firearms)? I personally know a guy who beat off about 10 people, putting several of them in hospital and he is not even a martial artist, he is just a strong, fearless guy.
Or say a Navy Seal, fully equipped with guns, ammo and armor finds himself transported through time and needs to fight 2 dozen of cavemen. I'm not saying that he cannot absolutely not lose. Like you said, many things can be changed by luck but a damn 1d20 is not an indicator of luck. According to it, the Navy seal can lose 9 out of 20 times, 3 out of which would be a great loss while 2 out of which will be tremendous loss, completely wrecking his shit. For the Seal to lose in that scenario his the odds should be astronomically high in number, not 1 out of 20.
Okay, lets have a no combat scenario. Imagine a carpenter, his father was carpenter, his grandfather was a carpenter. Since he was 6 years old he started learning tricks of the trade. He went to carpeting school. Entered Carpeting university, got a PhD in Carpeting, spent 20 years working as a carpenter. Now imagine we're playing that carpenter after all that and we have to hammer in a nail. Would you have us roll for that? 1d20 where there is 9/20 chance he will fail to hammer the nail and 1-2/20 chance he will fail so hard he will break his fingers? This is the situation that Shun keeps finding himself in.