Hi,
Like I said earlier, I don't have any experience with that piece fo code but if I'd venture I guess (and I'm pretty certain I'm right) it scans the pins once each time you GOSUB the routine. This means it does NOT scan the pins during the time you GOSUB the encoder routine.

Does Proteus show that walking zero across the pins at the same time as you do newState = PortB & %00000111 - I'd say no.

If your inputs are pulled up with resistors then the common/wiper of the encoder must be LOW at the time you read the status of the port. You must understand and remember that the PIC executes the code sequentially, one instruction after another.