Ok, I hate it when this happens because you look like such a fool!

UCFG.3 = 1 (this turns off the USB Transceiver which you also have to do to enable RC4 and RC5 as inputs.)