Maybe you could use a Timer to mask recently pressed buttons. When PORTB.4 IOC triggers, disable the IOC Interrupt for that pin, start your 50 ms Timer, then in the Timer ISR restore all IOC Interrupts, regardless of which one you disabled for the mask.