That would gain about 2.5 seconds per hour, which is not good.
Have you set LP oscillator mode in your configs?
Since it worked for you at 4mhz, it's probably set to XT and over driving the crystal.
How are you measuring that frequency?
Is it a high quality frequency counter?
Accurate measurements of low frequencies can be difficult, with large errors.
Yes, I change it to LP when driving it with 32 kHz and your advised process to run timer 1 from internal clock (loading TMR1H=%11100000) & XT when 4MHz (with 32kHz attached 4 TMR1).
For frequency, unfortunately I am not equipped with those luxuries, I am just using an Timer Calculator I found in the Forum only. Screenshot attached.
___________________
WHY things get boring when they work just fine?
Bookmarks