The Wikipedia page on the Southern Shaolin Monastery puts it beautifully: "The Southern Shaolin Monastery is the name of a Buddhist monastery whose existence and location are both disputed. By tradition it is considered the source of all southern Chinese martial arts. ...
The following account is based on legend or folklore, with little, if any, documentary evidence to support it."
By definition, we're talking about a folklore or oral history, about which genuine on-the-ground facts are scant, and the "things we know about it" inherently suspect. If there's anything I've learned studying Chinese martial arts, it's that derivations and origin stories are always thus. They are always filtered through layers of myth, imagination, yóuxiá and wǔxiá stories, kung fu cinema, and latter-day appropriations of all of the above.
Shàolín isn't a brand of martial gōngfu, it's a forest. Specifically, the one on Shǎoshì Mountain, where the (northern) Shaolin Monastery is located. By legend all kung fu emanates from that one source, spreading to the south and seeding Southern kung fu styles. Is that how it really happened? I don't know. I don't think anyone really does. Is it possible there was some transmission of styles, moves, and teachings, just as the "monks sent to combat pirates, and then stayed on" legend suggests? Sure. Maybe even likely. People travel, and they transmit their skills and learnings as they go. But is it likely that, as the legend says, all southern fighting styles originate there? Not really.
Humanity existed all over China both before and after the supposed monk's journey. And if we know anything about humans, it's that they like to fight. They fight for sport, for dominance, for riches, and everything else. They get good at fighting, forming gangs, tribes, and armies all over. They develop techniques, styles, and related sporting events. Just in the realm of folk wrestling, for example, there are dozens of styles from every conceivable place. Many of their moves have great commonalities--primarily because people have the same human bodies and must obey the laws of physics, regardless of where they're from. So it seems much more believable that most Southern kung fu styles emerged from parallel development, perhaps accelerated by some idea diffusion and technique sharing by travelers, than that there was just one source, which fanned out hierarchically.
The "it all came from Shaolin Monastery" idea is also hard to credit given the styles that resulted. I've studied both northern styles (běiquán) such as Chángquán/Longfist and southern styles (nánquán) such as Five Animals. There are definitely some similarities. The "Pull Sweep Push Sweep" of Dragon, for instance, can also be seen in Longfist's "13 Tripping Methods." There are probably a dozen other places you could say "they have the same move." But overall, the styles are very different. They focus on different ideas. They move differently. They have different explanations of why you'd move in a given way, and what you're trying to accomplish. Maybe there are other southern styles with closer linkages. Wing Chun has a lot of the same linearity, for example, but it too moves very differently, using different controls. No southern form I know of, at least, appears directly linked to, or imitative of, the northern styles I know.
These points are suggestive, not probative. But before diving too far into the mythical trek of kung fu from its unique birthplace, it makes sense to ask "Is that even possible? If so, how likely? And do the styles that resulted show any evidence?" From this vantage, it's looking like at least 90% myth.