Are you setting oscillator selection bits in config word for HS?

A 20MHz crystal requires a higher drive level output on osc2. Make sure you
set config word for HS instead of XT.