I had one last thing to test:

The slide pot was soldered onto a new PCB from JLCPCB. It's just the pot with a connector, nothing else to interfere (I do this to test all my devices).

I had used a 3-pin header to connect to the breadboard (brand new, very tight, no jiggleness in connections).

I tried again, but this time using a female header socket, and used jumper wires to connect straight to the PIC on the breadboard.

Same jitters, so I've eliminated the header connector as possible issue (the PCB only holds by those 3 pins, so it had some
movement).

Looks like I'm down to stripping the bottom 4 bits.

One last possibility, finding a PIC on JLCPCB with at least 8 pins with 12-bit ADC. Not my favourite idea; I was looking at using the TQFP-64 16F1937 for my LEDs, switches, pots and encoders. If I have to buy PICs in bulk later, I wanted to limit my use to 1 for USB, 1 for I/O to limit costs.