The pot has a reading of 2.0mV when full right:
each adc step is 4.88mV therefore it should read zero

you get a reading of 12 , after un-dividing by 4 that's a count of 48 or 0.234 volts
it virtually has to be power supply noise.
no amount of dicking around with reference voltage's is going to eliminate that much noise

eliminate your power supply as the culprit by using 3x fresh AA batteries in its place