Assumptions and Background
Let's assume we have an impossible situation: two people who are identical except the aforementioned difference in weight and conditioning. Additionally, let's assume these two are aware of each other's intent. No getting the jump on the other guy!
Additionally, I am also using my experience with Historic European Martial Arts and other sports to inform my answer. That covers a lot of things, such as wrestling, knife fighting, spear fighting, all the way through to mounted sabre combat of WWI.
Factors
As you have identified, strength and agility of combatants are major factors in combat. People have recognized this for a long time. People have also recognized that these are far from the only factors.
In fact, Fiore di Liberi, a 14th century swordmaster, adds two more fighting factors/virtues. Addmittedly, people translate his values as "prudence," "celerity," "audacity," and "fortitude." There is an argument for translating then as "skill," "agility," "bravery," and "strength."
Other significant factors exist, like terrain and awareness. In reality, these factors are hugely important. Our improbable situation has isolated for this.
My Money is On The Big Guy
Why? Simply because of the extra mass he has to lug around. Sure, the small guy is fit, but this is unarmored combat. As you assumed, the big guy will likely be able to take a lot of punches and kicks. The extra weight and strength is just too important when it comes to unarmored fighting. Boxing and similar sport-combat events have weight classes for this exact reason!
This being said, the small guy's most likely route to victory is "dirty" fighting. He needs to yank/crank joints, use arm and body locks, and generally rely on the weak points of the human body. Odds are that his open-handed martial arts training has informed him of these and his athleticism will allow him to get where he needs to be.
Is There a Clear Winner?
I like the Odds for the Big Guy, but upsets happen. If we ran this many times, I would give the big guy(s) about 70% win rate. It's an uphill battle for the agile guy, but the big guy needs to take him very seriously.
All this to say martial arts are arts: the art is in making correct/good choices in the heat of combat. Too many things happen too quickly for this to be a chess match or be anywhere close to deterministic.