It did toggle with my previous setup with switch on B.0 using INT_INT so it should toggle now on B.5 using RBC_INT as well.

When I press the switch I get 2.2V on the led

I think the problem is due to some noise on the lines as my led fill go on/off when I touch the wires. Never had it on B.0, the toggling was smoth with no noise and no dimming.