I usually just wear my glasses with an elastic strap to hold them on. That's enough to keep them from flying off when moving quickly. I occasionally take them off, particularly when they get too messed up by sweat, or when I forget to bring a band for them, but I find it difficult to follow the instructor when I can't see clearly. I generally don't worry about them breaking from getting hit because anything that hits me hard enough to break the glasses is likely to hit hard enough to damage my face as it is. I have had one or two people express a worry that the person striking might get hurt, a near miss snagging the corner of the glasses and tearing skin, but I haven't actually had that be a problem.
In regards to contacts, I have heard about increased risk of retinal detachment when wearing them, but I don't find any current references to that and being nearsighted increases the risk anyhow. Probably best you don't wear them.