There is no problem with weak pull-ups on switches...

How are you enabling them?

Apart from OPTION_REG you also have to set them in the WPUA register in this chip.