Using the same here, but tested it on real hardware with a pull-down holding RB3 low, and
switch pulling it high when pressed. Works 100%.

Is RB3 being held at ground until you press the switch?