Hi,
I don't have any experience with that routine but provided everything else works the way you want why don't you just set the pin which is common to the encoder (RB5 ?) high, read the port, and return the pin low.

GOSUB ScanMatrix 'Or whatever
PortB.5 = 1
GOSUB Encoder
PortB.5 = 0

Finally, in the first code you posted you had the encoder to RB7-5 so I based my code on that. Now, in your latest schematic, you have it on RB2-0 which means that my code (if that's what you're trying) definitely won't work without modifying it since it expects the three "switches" in the encoder to be on RB7-5.