Try adding a bit of capacitance from the wiper of the pot to ground - say 100nF.
This should minimise some of the variations you are experiencing.

Cheers
Barry
VK2XBP