I get severe problems with my back and joints. For example, I was training rigorously five to six times a week for about five months while preparing for a grading test. Immediately after, I went on a relaxing holiday, where I spent most of the time in stationary position. A week into my holiday, I got serious problems with my back – as in, I couldn't walk or even sit up straight. This has happened before, and now I try to keeping active during my holidays. I get similar problems with my knees and elbows (although this may be related to my hyperflexibility).
I've heard other similar stories. If they practice hard for a number of years, and then suddenly stop, their old injuries will come back to haunt them, because the muscles are not there to support any more.
Apart from the physical problems, I also experience mental issues. I get very grumpy, tired, demotivated and agitated – it also affects my sleep, like dmckee mentioned. This makes sense, because exercise releases endorphines, which makes you happy and "high". This is why people who suffer from depressions or stress are often told to exercise more. When you stop exercising, you experience withdrawals, just like with cigarettes or drugs (although in a milder form, of course).
I know this is not the "correct" scientifically proven answer, but it is my personal subjective experience.