Well, look in the datasheet, Oscillator section, and check the available configuration fuses setting. Yes they're a bit tricky to understand...

From there, you may discover that you can even have a processor speed of 48MHz using a 4MHz crystal... this is accomplished by the internal set of clock divider and PLL.

As Microchip's bootloader set the Config Fuses for an internal processor speed of 48Mhz, your code must follow it.