You're a female, and you're mostly interested in self-defense. Presumably this is because you just want to be able to defend yourself in common real-life situations women might find themselves in. And you're worried that the class you enroll in will only teach things that are useful for much taller men.
Briefly, my recommendation for you is to look at Brazilian Jiujitsu. Specifically, if you can find a Gracie style BJJ school, that's where you should go. Gracie Jiujitsu has a long track record of success with regards to training women. So does Judo, but Jiujitsu is more oriented towards practical self-defense.
Regarding physical attributes (gender, height, weight, strength, and athleticism) and how it affects your ability to defend yourself: Generally if all of physical attributes are equal, the person with the most skill will have a better chance of "winning" in a physical confrontation. When there is a physical attribute disadvantage, greater skill is required.
Keep in mind that those physical attributes do matter. A black belt in Brazilian Jiujitsu or any martial art will have a harder time winning against a 200 pound white belt than a 150 pound white belt. Size does matter. There are even cases where BJJ black belts lose to beginners, just because the beginners are able to use their overwhelming strength and weight to their advantage.
But skill also matters. Skill can make up for size disadvantages. And generally speaking, the more the size disadvantage, the more skilled you have to be to overcome your opponent.
What that means for women who want to learn self-defense in preparation against men is that they will need to train for a longer period of time to be able to reliably defend themselves against larger men. So the average man starting Brazilian Jiujitsu training may only take a year to reach the point where he's skilled enough to reliably defeat other average sized men. But the average woman starting BJJ may need two or three years to reach the point where she's skilled enough to reliably defeat an average sized man. It just takes longer.
This physical attribute disadvantage is inherent in any martial art. If you find a martial art that tries to tell you size doesn't matter in a fight so long as you're using their martial art, just walk away. They're either very delusional about their martial art or are trying to sell you on it. Don't waste your time or money there.
If you have a choice of BJJ schools in your area, go to each one and check them out. You should make sure the school has women already training there. Many schools of BJJ actually have a separate women's-only class and allow those same women to participate in coed classes as well if they want. Make sure whatever school you choose, it places self-defense higher than the sport aspects of BJJ. That's why I say I prefer Gracie Jiujitsu for this, because they try to ensure that self-defense is their core material, and sports is secondary.
I wrote a lengthy answer in the past that describes exactly what "realistic" self-defense practice should look like and why. It was to a question about someone asking if Taekwondo was good for self-defense. I think I'd just be repeating most of it to explain to you what you want to be looking for, so please read my other answer at the link here:
is Jun Chong TKD a legitimate TKD dojo for self defense?
(Skip to the part which begins with "Moving right along...".)
Alternatively, I'd suggest looking into Krav-Maga. You can take a small course about 3 months long usually, and they'll cover a lot of practical topics. It gives you a good introduction to what it's like to punch, kick, elbow, and knee people. It teaches some throws, some grappling techniques, situational awareness, and maybe even some weapons stuff. Its ground fighting training will be pretty minimal, though. All of that will be in the first 3-6 months. It's designed to kind of give you a crash course on unarmed combat. So from that perspective, I like it. It doesn't have all the extra baggage that comes with traditional martial arts training (there are no kata / forms, no meditation, no bowing, no yelling kiai, etc.).
But personally, I think you'll be able to take more from Brazilian Jiujitsu that you can actually remember and use successfully. I say that because of the way BJJ schools train people: with fully resisting partners (but with safety as the highest concern) that don't let you fool yourself into thinking you're able to use it for real. Krav-Maga might give you a small taste of that kind of training, but it's not done nearly as often.
That's the thing that really counts when learning realistic self-defense, more than anything else in my opinion. You have to know it can work on a fully resisting larger man first, in class, before you'll ever have the confidence that it will work for real outside of the class. Most martial arts schools don't give you that. Instead, they give you a false confidence, which can get you killed, or at the very least, it will waste your time and money.
Hope that helps!