Depending on your chosen style, going for a 90 degrees angle right from the start may not be the proper position for the horse stance. In shorinji-ryu karatedo, for instance, we usually go for about 45 degrees with the knees about shoulder width.
Although we eventually end up going lower as a training exercise to build leg strength, we don't advocate doing so for lower level students. In fact, most of the beginners in the dojo try to copy the lower stance of the higher belts and end up bending forward themselves. When we notice this, we set them on a slightly higher stance, closer to 45 degrees, and their back goes a lot straighter.
I'm no physician, but I assume this is due to better postural muscles in the lower back. Notice how your weight is distributed when you get in horse stance. Normally, your center of mass should be a little behind the imaginary line connecting your feet, as you seem to have noticed yourself. The horse stance is not naturally comfortable and, if you go too low too fast, you will bend as a way to maintain balance, especially since your own center of mass is pulling you to the back.
If you try to ease into the lower stance over time by starting a bit higher, in a stance that, while not as strong, is definitely more natural to your body, you will develop better postural muscles that will allow you to keep your balance more easily. These postural muscles, probably as much, if not more, as those in your legs, are what will eventually give you the feeling of being rooted into the ground (figuratively speaking, of course). This is when you know your stance is strong and you become very difficult to move around.
Start with the lowest you can go that allows you to maintain balance with your back straight. Train like this for a few classes, then try to go a bit lower. As soon as you can do so without losing balance and still maintaining proper posture, bend a bit more. Don't try to rush it, as you might develop bad training habits that will make your stance weaker in the long run even though it looks good at first glance. Keep training for a bit and, after a few weeks or months, depending on your training schedule, you should see vast improvements in your stance.
Personally, I developped a really strong horse stance by cooking. I kid you not! I have a small kitchen, so when I cook, I don't move around much. So instead of just standing there cutting vegetables and stuff, I used to drop into a horse stance for as long as I could hold it. I eventually got to the point where I could handle being low enough that it wasn't very practical anymore, as I got lower and the kitchen countertop didn't.
It's normal to bend forward if you go too low too fast as your postural muscles aren't accustomed to this stance. Start a bit higher and gradually lower into the proper stance over a few weeks/months. You'll build the proper muscles over time and your stance will become stronger and lower. Tip: practice the horse stance everytime you'd just be standing up for another task.