Code:
 If UpButton THEN
    PAUSE 75
    Setting = Setting + 1              ' Increment setting
    If Setting = 6 THEN Setting = 5    ' Make sure it doesn't go beyond 5
    SettingChanged = 1                 ' Set flag indicating new setting
  EndIf


  If DnButton THEN
    PAUSE 75
    Setting = Setting - 1
    If Setting = 255 THEN Setting = 0  ' Make sure it doesn't go below 0
    SettingChanged = 1                 ' Set flag indicating new setting
  EndIf
Adding the PAUSE 75 fixed the run away issue and gives a quick response to a button press while still being easy to use.

Still havent figured out why at DutyCycle=0 there is still a very low pwm signal that lasts for just a few uS. Or while at setting 5 it rolls over to setting 1.