If only RB3 causes your problem, then I suspect you're simply not getting LVP turned off.

It would really help if you posted your code.