Hi,
Without analyzing your code in detail one possible problem might be this line
Code:
Q_New = PortB.7 + PortB.6 + PortB.5         ' get port status
In this case you'll get Q_New = 1 no matter WHICH of the three pins are high. I'm not sure but I guess that's not what you want.

You could try something like Q_New = PortB & %11100000 >> 5 instead.

Here's another approach (untested) which is smaller in terms of program space and RAM but like I said it's untested so it might not even work.
Code:
oldState VAR BYTE
newState VAR BYTE
Q_Count VAR WORD
DIR VAR BIT

UP CON 1
DN CON 0

newState = (PortB & %11100000)      ' Isolate top three bits, newState will be 32, 64 or 128 

If newState <> 0 THEN
If newState <> oldState THEN        ' Changed from last time?

  Select Case oldState
    Case 32
    If NewState = 128 THEN          ' Was 1 now 4 = Up
      Q_Count = Q_Count + 1
      DIR = UP
    ELSE                          ' Was 1 now 2 = Down
      Q_Count = Q_Count - 1
      DIR = DN
    ENDIF

  
    Case 64
    If NewState = 32 THEN        ' Was 2 now 1 = Up
      Q_Count = Q_Count + 1
      DIR = UP
    ELSE                           ' Was 2 now 4 = Down
      Q_Count = Q_Count - 1
      DIR = DN
    ENDIF

  
    Case 128
    If NewState = 64 THEN        ' Was 4 now 2 = Up
      Q_Count = Q_Count + 1
      DIR = UP
    ELSE                       ' Was 4 now 1 = Down
      Q_Count = Q_Count - 1
      DIR = DN
    ENDIF

  END SELECT

  oldState = NewState

ENDIF
ENDIF