The problem was fixed by adding this to the configs
_XINST_OFF_4L
And I made sure the internal regulator is on to feed the internal USB pullups.

Power wise I am running the chip at 3.3 so the only thing I have connected to VUSB is the capacitor to VSS.

Now I do have some more reading to do as there is mention in the data sheet about VDD has to be higher than VUSB.
But it is working for now. Long term ????

Newbies---- IMHO Compared to two or three here we all are some newbie. But we can learn from anyone if we care to listen.