I want to make sure I hit every part of this, so I'm going to make heavy use of quotes here...
First, we need to answer an unasked question: What is pain?
This is important first off since, in order to work through pain, you have to understand pain. Pain is a nervous-system response to intense or damaging stimuli. When someone strikes you, your body perceives the stimulus, nervous impulses fire, hormones flood your body, and your brain sends a message – Something has happened at XYZ location, be aware of it!
Part of overcoming pain is overcoming fear. The fear of pain intensifies pain and prevents you from thinking through the problem. This does not help, so it can also be important to experience pain as a means of overcoming a fear of pain.
Finally, it's important to notice that it's the brain that sends the impulse as to how pain is perceived. There are those who, either through an addiction to the hormone release, or through mental control, or even through misfiring of neurons experience pain as a pleasurable sensation. Since the brain tells the body what to feel and how, we have an opportunity to change our interpretation of that pain, maintaining acknowledgment that it is a warning of imminent danger, but perceiving it as a welcome message, rather than a crippling one.
Now let's answer your questions.
How do you train attacks/defense while in physical pain?
The problem that most practitioners experience is that they jump straight into pain. That is, from the first moment, they're taught to feel pain, to tap out, and the pain ends. This instills a fear of pain. Like a child learning to walk, you have to take easy first steps, not attempt to run full out.
First, experience pain.
When I was first faced with this same issue, my shidoshi (instructor in the Bujinkan) was using me as the class uke, demonstrating techniques on me with regularity, and always taking them to a final control. After a bit of this, he would make the pain excruciating; I would be screaming in pain. Eventually, I knew that I had to take the pain, and I began to "drift off". I would use visualization to get myself into a nice place (I have a fondness for Maui, so I'd visualize myself on the beach there), only to be wrenched back to reality, being told I needed to experience it. He was right.
Once you experience pain, you can isolate the pain. Use the pain, and feel for it; what direction is the pain moving, how are you contributing to it, and, most importantly, what happens when I breathe?
A French obstetrician by the name of Dr. Fernand Lamaze, in the 1940s, became rather fascinated with the techniques practiced by midwives in the Soviet Union. Many of these practices included breathing and relaxation techniques guided by the midwife during childbirth, and led to the development of a series of techniques designed to increase a woman's confidence in her ability to give birth. Similarly, these techniques are used by (and may stem from) Cossack practices for avoiding or shrugging off pain, by giving the mind something else to focus on. The human brain is fairly well maxed out most of the time on its ability to multi-task, and forcing a specific focal behavior (e.g. breathing) can be distracting. Distraction, as you'll note I'd already found, is imperative in blocking pain. Finding a distraction that allows you to stay active and focused is far better than imagining yourself on a beach in Hawaii.
When you find yourself in pain, first start breathing in forced deep breaths. While you do so, find the source of your pain. Once you find the source, the most general rule of escape is to move toward the pain. Note the way you need to move in performing escapes from shoulder locks, wrist locks, chokes, etc. when you're training escapes. Pain wants to draw you away from the thing hurting you; sometimes to escape a burning house you need to leap through flames.
Are there safe ways to cause a lot of physical pain without long term damage?
Well, yes and no... Any impact injury carries risk. Any rending, tearing, pulling, grabbing, etc. can have the risk of dislodging a blood clot leading to infarction; grappling could lead to fracture and bone shards could cause an embolism. That said, so long as you're carefully training, these risks are minimum.
Training with a partner you trust is key. Exploring your pain threshold carefully can lead you to a greater understanding of when enough is enough, always being careful to push just past where you want to tap. You want the immediate result, but there is no easy way to learn to overcome pain. It's equal parts personal resolve and psychological trickery to move past the things that cause you intense pain.
How do you learn to recognize when there is too much physical pain?
Very good question. Before you begin training to resist pain, you need to learn what causes you pain. Work with a partner in slowly inflicting pain upon each other, being careful to not injure the other, and resolving to push further than you'd normally allow before tapping.
Generally, if you can't escape from this level of training, then you just need to tap. Your partner should continue to slowly apply greater pain until you tap or escape. If something gets torn, you've gone too far. Careful training is vital, and only you can know your limits.
Here are some examples of pain that I have experienced but was unable to overcame as quick as I'd like:
This is irrelevant. All pain offers the same effects. The compression of the sciatic nerve leads to a collapse in the legs, compression of the suprascapular nerve leads to a numbing of the arm. But pain is pain. These nerves carry impulses, and you can not overcome their compression immediately; it's a physiological function. The pain you feel, however, you can overcome, but to do so quickly will mean practice. Nothing comes overnight.
For those that cause a momentary loss of function, train from the position to which you're dropped. Do that, and you'll still be defensible.