robert states
Tried shifting out low 4 bits.


...mostly stable
that means the pot resistance must vary in leaps of over 160 ohms [assuming correct technique]
no amount of lpf will be useful in getting stable readings from "fine" movements under those conditions