This actually happens quite a bit. I don't have statistics on how often it happens, but there are a ton of martial arts schools headed by people who essentially started their own school and declared themselves a 10th dan grandmaster. They may have something worthwhile to teach, or not. But because of the competitive nature of martial arts schools, few of these systems continue running long enough to train someone to head the system after the original founder retires or passes away. When that happens, the systems just sort of fade away. Students move on to other systems.
Another aspect of this is that systems like this are often headed by the kind of person who thinks he/she has some profound and special insight into martial arts, or they just want to feel important. It appeals to narcissists, in other words. They like having people looking up to them. And to keep it that way, they hand out rank at a snail's pace and change the goal posts over time. They want to ensure that they are the big boss forever, so they don't want any of their students to reach their level. They'll make up new stuff over time to make sure nobody can catch up to them.
I'm not saying that's what happened in your case. I'm just saying this is pretty typical in the martial arts world.
Regardless, what is needed in order to pass a martial art down to a new grandmaster is either: 1) The old grandmaster formally declaring it, in writing or verbally in front of all senior students. Or 2) If the old grandmaster didn't pass it down officially, then the senior students must vote to determine who becomes the next grandmaster.
In each case, there can be schisms. Some students reject the new grandmaster and start new schools with themselves as the grandmaster. This happens a lot. It's upsetting, but it's all part of life.
When a new grandmaster is declared, then that person typically becomes a 10th dan. Yup, he could be a 3rd dan today, and tomorrow he's a 10th dan after he becomes a grandmaster officially.
The thing is, though, chances are pretty good that the original grandmaster who started the style wasn't a 10th dan, either. Many of them just achieved low dan ranks (1st, 2nd, and 3rd dan) in one or more other martial arts before forming their own system and declaring themselves as 10th dans. It's the rule, not the exception.
There's really nothing stopping anyone from declaring themselves as grandmaster of their own system, by the way. You don't even have to wait for the previous grandmaster to hand it down to you. And you don't even need any martial arts training!
As for your particular case, first I would declare myself "acting head instructor" of the style if there was nobody higher ranked than you. And I would continue teaching the school that was left behind. Find all the former students and invite them back to train with you.
You mentioned there were previous students of your grandmaster who have since left the system but were higher in rank than you. If I were you, I would write to them all, tell them what happened, and ask if they would meet with you to teach you whatever was next in the system. Gain as much as you can from them.
If your grandmaster had a family, I would approach them and ask if you could have any of the notes, drawings, videos, etc. They would probably just throw them out and would love that you're willing to take the collection and continue his legacy. Absorb whatever you can.
Then, after some time has passed, maybe a year or two, ask all your grandmaster's previous senior students and immediate family to submit their nominations for the new grandmaster. After all nominations are in, hold a vote. Maybe have a run-off election between the two with the most votes. And then you have your new grandmaster.
That will probably be you if you're the acting head of the system now. But you never know.
The new grandmaster gets 10th dan. There should be a big ceremony and banquet with every student invited and especially the family of the previous grandmaster. That will be when the new grandmaster is officially declared. A local news reporter should be invited as well, to document it in the newspaper.
If you're the new grandmaster, it means you have some work to do to take the style further than what you learned. You should make it your life's work to go out and learn what else is out there, whatever can advance your understanding. An obvious direction to take is to study from your grandmaster's instructors.
Your grandmaster is no longer here. It's no longer his system. It belongs to the next grandmaster. And yes, that means it might go in a totally different direction. That's okay.
Hope that helps.