Sugar just disrupts the pH balance of things, i.e. the pH in your mouth is generally 7.5, but when you eat candy, it drops to 5.5, allowing bacteria to grow and make cavities...Think of how delicate your skin is compared to your teeth.
I use brown sugar to scrub my face now and then, I used to do it more frequently, until I realized it was causing damage, and making the problems worse.
Also, sugar is the major cause of yeast overgrowth, which has been linked to a number of skin problems, like acne, rosacea, atopic dermatitis, 'chicken skin' etc. etc.