I figured out what my biggest problem was. It was a hardware problem. It was SW3. The on-off-on toggle switch. I should have had the pole connected to a voltage divide. For some reason, it worked the way it was supposed to until the pot was below 20%. I don't know why the position of the pot made a difference, but it did. Anyhow....added a voltage divide and it works perfectly.

The missing voltage divide aside, the pot still needed to be "debounced". Thank you so much, Richard for the massive improvement you made to the way the ADC reads the pot. Thanks for your help too, Henrik.