My dermatologist never mentioned this, and he had a lot of rules for me to follow about what products are safe for acne-prone skin! (haircare, makeup, etc.) I have a lot of acne info in my notepad if you're interested. How could polyester cause breakouts? Do little pieces come off and clog your pores?
Now facial oils can build up on your towels and washclothes and then clog your pores, so you'll want to change them often. I change my washclothes daily (you can get a pack of cheap washclothes at Wal-Mart if you don't care about colors!) and have a towel that I only use on my freshly-washed face which I change weekly. (Similarly, facial oils can build up on pillowcases, so I change those at least once a week.)