The fact that Wladimir Klitscho could hug his way to win after win keeps me up at night tbh...
Clinching is difficult to manage as a ringleader. That's why, I assume, it goes unpunished. At one point, you need to allow infighters to infight(meaning when 1 hand is available, even though the other is holding - for example).. As per why it isn't punished? It's definitely a good question, and one we couldn't answer. Boxing has rules, but often follows what general consensus is instead of what's present in the rulebooks - it's a funny sport in that regard.
To answer your questions..
So if it is illegal, why are penalties so rare (in pro fighting)
Penalties are, in general, very rare. Good ringleaders(for world championship fights, of which you, a spectator, is more likely to see) are not there to get attention. On the contrary, they want to intervene as little as possible and let the fighters fight. So penalties are, in general, just rare. Warnings are common, though.
and if it is necessary, why is(hugging/clinching) illegal?
Because it resets the pace of the fight. Anytime someone gets too close to Klitscho, he will hug them and let the ringleader reset, so he gets back to his comfortable reach advantage. As you can see, in a lot of heavyweight fights, these are snoozefests because it's 1-3 seconds of punching, and 10 seconds of hugging/man-handling each other. It's ridiculous and I completely agree with you that it's surprising nothing is done.