REMOVE IT!
I had a rather large and deeply rooted mole on my chin that I felt so self-conscious about for years. I considered removing it but various dermatologists said it would leave a scar and it was better to just leave it because the mole was flat but deep. Well my regular doctor advised me to get it removed, by a plastic surgeon, and I listened. He did a fabulous job and I am so happy I had the mole removed. The scar is barely noticeable and it's healed so well. it's been 2 years later and you don't even notice the scar. Plus I feel better about it, even though it was just a mole.
Basically - if it bothers you enough, get it removed. Just make sure you go to a good doctor, especially since it's on your face.