I use a DC-DC converter to lower 12V down to 5V for the entire circuit.

Would a dedicated 7805 for the pots be a "meaningful" improvement?

I'll have up to 10 pots in the final circuit. Right now I'm trying to get 1 ADC working well.