I think you should read the datasheet. The chips with USB have a wide variety oscillator PLL and divider configuations. Your chip is obviously not runnng at 20Mhz - internally, even though the oscillator itself is running 20.