the default config is for a 20mhz xtal which gives a fosc of 48mhz

to use a 24 MHz xtal change plldiv to 6 for a 48mhz fosc or turn the pll off .
24 mhz into the pll @ plldiv5 is out of device specs.
with pll off and cpudiv=1 fosc=24mhz fo a 24mhz xtal.