There are a couple of ways that a direct blow to the chest could cause damage, other than the freezing of the diaphragm from a blow almost anywhere along the line of the ribcage (Although the side edges and solar plexus are the more vulnerable points).
Breaking a rib
If you can punch around 3300 newtons of force, you have about 25% chance to fracture a rib, according to a Livescience article. A typical boxer can generate up to 5000 newtons with a punch. A fractured rib will impair both breathing and movement immediately, and depending on the break, a chance to rupture blood vessels running along the rib itself. (This also depends on angle of strike, bone condition of opponent, many factors. This is just a rough example).
Stopping the heart
A direct blow over the heart has a chance to stop the heart. The basic mechanism is that the blow strikes during the major repolarization phase of the heartbeat (Called a T wave on an ECG), and disrupts the rhythm. This is called commotio cordis and while rare and more prevalent in youths, it does happen in many sports, and has a mortality rate above 60%.
Why the chest?
As pointed out already, striking the face is potentially very harmful to the person doing the punching. Additionally, a punch to the face may do no more damage than a nosebleed and really enraging the opponent. Knocking a person out relies more on torque causing the brain to bang against the skull. There is no guarantee that you are going to be able to do that with a punch.
Add to that, the head is relatively small in area compared to the chest, and can easily be covered, or deflect blows coming at it. In comparison, the chest has a very broad area from front to back and top to bottom, and it takes a skilled person to guard that.
The final point - Against an untrained person, the typical reaction is to jerk the hands down in front of an incoming punch. An experienced fighter can target the ribs, potentially do damage and cause the hands to move, then target the face if desired once the guard is moved.