Yes and No.
If I am close enough to you I can move my shoulder and punch you long before you can dodge or block. If I am far enough away I couldn't possibly hit you without your reactions giving you a good chance to evade my strike.
In addition many styles, boxing included, will teach ways to hide shoulder movement with foot movements and clever steps.
That said, what are the alternatives?
If they are bare handed then you can watch the forearm and judge the position of the fist that way. But if your opponent is wearing gloves you are not going to be able to judge the position of the hand easily because the glove will hide the forearm so in this case watching the shoulder might be a good plan.
Alternatively if you are really good at pushing-hands type drills you might be able to maintain contact with your opponent's arms at close range. That way you don't rely on sight at all.
In all of the above you have to consider the context of where you are fighting. If you are in a self defence situation of some kind it probably doesn't matter if you watch the shoulder or not so much. After all your goal should be survival not killing anyone.
As soon as you know that your enemy is trying to harm you you should be trying to think about exit routes and if they are close enough to punch you at all you have already failed in your best line of defence. In other words, watch for the monkey dance and the shouting not the first punch.
If you are in a ring watching the shoulder is more relevant. You are going to fight the guy, probably to unconsciousness. So this is where you actually have to think about their 'tells'. In this situation you should have watched videos of their fights and know what to look for anyway. The shoulders yes, but often also the mouth, the eyebrows etc.
In conclusion, reading a fighter is a skill in itself. There are no shortcuts, but watching the shoulder is one option.