You'll also need to disconect the GND line then. When you apply power to your PIC, the Hi-Z is granted on the output port, while not when you disconnect the power...