At least for the 12F683 it has a charge holding capacitor in the ADC circuit. The way I understand it that needs to be give the appropriate amount of time to charge fully before you take a reading. That time may change depending on which chip you're using.

My guess is that Demons suggestion of a short pause would solve the problem. Or a counter that won't allow display until the appropriate time has elapsed. Either gives the ADC a chance to stabilize and provide the correct reading.