It's hard to say what's happening without seeing the code.

Keep in mind that crystals do not keep perfect time. They have a rated accuracy - usually in ppm (parts per million). They are also prone to changing frequency with temperature and time.