Mine don't refresh the PWM duty cycle all the time, it will do IF a push button is pressed.

The main difference is how the switch debounce is made. Let's use a Google Image

This is what the signal looks like when you press your switch, the same noise should also appear on the rising edge when you release the switch.

A little pause after you press on the switch is probably not necessary because your code need some time to execute, but 20mSec for safety sake that's nothing huh?

If you had more code after the switch detection, you could reduce the second PAUSE.

HTH