And...
The OSC is not running the chip, it is running the PLL, That is why the 48MHz works.
With the correct config settings you can run at 48MHz with a 4MHz external.

Another cool thing is the different voltages this allows. Normally you have to use 5 volts to run a chip at 20MHz, with a 4MHz and the correct configs you can run this chip with 2 volts at 48MHz. For USB though you will not want to go lower than 3.3 volts, so that way the whole system can run at "low" voltages at "high" speeds. Really nice when using things like SD cards, some RTC, and other things that only run on ~3 volts.