i don't feel that 20MHZ clock will be fast enough to provide any USB communication. 40MHZ would work so, you can even use a 10MHZ crystal and set the HSPLL to on. That way, it 'should' work.

But is PLL work great and bug free for that... i can't tell.