I'd be more worried about ear-sweat and other build-up. Your ears need ventilation or they can get infections or other bad stuff. I know whenever I use earplugs I get a lot of wax build-up. Neil Peart had an ear infection because of too much use of in-ear monitors and motor-bike ear plugs.
That said if you can't get to sleep any other way it might be worth the risk.