I am not sure if my comments will help. I am putting in some check points for you to look at.

Is it possible you are reading the ADC too fast / very frequently ? Usually, you need to allow a settling time between setting the ADC channel to read and acquiring the channel value. If that is not handled correctly, you may see such problems that you describe. usually, a 1 or 2 count variation on an averaged value is reasonably steady for a 10bit adc.

If it is not in firmware (since you say so - the rotary pots do not exhibit such jitter), you could possibly look towards the hardware. Are the pots you compare - rotary and slider - of the same value? Are they both LINear or LOG types? A review of these will help

Thridly, you should ensure that your grounds do not form what is known as a ground loop. This can disturb the ADC acquisition if the ground currents change the measured voltage by a few mV (for 10bit @5V, it is approx 5mV for LSB). So, ensure that any heavy currents are properly routed to the power supply. Examples of such current - 7 segment display currents passing through a sink device (maybe a ULN2003 or such) whose ground is connected to the CPU which then connects to the power ground. The current flow should ideally be from CPU ground to ULN2003 ground to power supply ground. To identify such a ground loop, I would connect a fly-wire to the power ground and then touch various grounds in the circuit to identify where the error is mitigated.