Yes.
It's pretty obvious, isn't it? It's why there are weight classes in all competitive fighting sports.
The Gracie family says that for every 20 pounds heavier, you need to be one belt higher ranked than your opponent, all other things being equal. The "20 pounds" figure is not to be taken literally or scientifically. It's just a general concept. There have been cases of white belt men weighing 250 pounds beating black belt women weighing 120 pounds.
All physical attributes matter in fighting. Your age, height, weight, sex, strength, speed, and stamina all matter.
Skill matters, too. But if you have two people with equal skill, and one of them is just bigger and stronger, then the bigger and stronger one usually wins.
Why this is even a question is because traditional martial arts have promoted the idea that all that really matters is skill. You see it in kung-fu movies all the time. The little guy is always beaten up at first, and then he meets a martial arts master who takes pity on him and teaches him the "real" stuff. In the end, he takes on his bullies and wins easily.
That's not the way it works out in real life. More realistically, a smaller, weaker person will learn enough over many years of hard training to survive a fight and get out of there before he gets seriously hurt. And it will be very unreliable until he gains much more fighting experience, which is typically not how traditional martial arts train.
It's nothing more than magical thinking and delusion to believe there's some secret, super effective technique out there known only by some true master, and if you just learn it and train with that master earnestly, you'll be more or less invincible. That's just a nice fantasy, something you can picture in your head as you're drifting off to sleep at night.
Hope that helps.