I am running the same chip at 48Mhz with a 4 Mhz osc at 3.3 volts.
The only thing about USB that is 5 volts is the positive power line. That has nothing to do with the signal. As far as I know even a USB flash drive runs on 3 something. Part of the stuff in the stick is a 5 to 3.3 regulator.