Did you set fuse PLLDIV = 6?
PLL expect 4MHz input clock. So you must set correct divider for your crystal oscillator.