That would gain about 2.5 seconds per hour, which is not good.

Have you set LP oscillator mode in your configs?
Since it worked for you at 4mhz, it's probably set to XT and over driving the crystal.

How are you measuring that frequency?
Is it a high quality frequency counter?
Accurate measurements of low frequencies can be difficult, with large errors.