Some devices have internal soft pullups on port B that must be enabled. I'm not sure yours is one or them - check the datasheet.