I had a similar issue a while ago maybe I can be of some assistance.
I was also using the F84A, on PORTB with the POT command. I noticed that larger pots, over 15mm dia, read very clean, yet the small PCB mount types can be noisy, with values moving around, especially if they are the cheap and nasty 0.20c types.

When using small PCB types or small cheap pots,I compensate with software. I sometimes read 5 values, add together and get and average read, or other times I will read only even numbers and ignore the odd. It all depends what you need it for...

Also check your common GND and make sure your regulator can provide enough juice for the circuit. Are you using 78L05?