Don't know what to tell you but the code works just fine here.
Code:
HSEROUT ["Program start",13]

Main:
    PortB.5 = 0
    newState = PortA & %00000111
    PortB.5 = 1
    
    If newState <> 7 THEN
        If newState <> oldState THEN
              
            Select Case oldState
                Case 3
                If NewState = 6 THEN dir=up         
                If NewState = 5 THEN dir=dn
  
                Case 5
                If NewState = 3 THEN dir=up        
                If NewState = 6 THEN dir=dn
  
                Case 6
                If NewState = 5 THEN dir=up        
                If NewState = 3 THEN dir=dn
            END SELECT

            HSEROUT["oldState: ", DEC oldState, "   newState: ", DEC newState, "   DIR: ", DEC DIR, 13]        
        
        oldState = newState
        Pause 1000
        ENDIF
    ENDIF
    
Goto Main
This was executed on an AMICUS18 board with a PIC18F25K20. A wire from RB5 to the pulled up inputs was used to simulate the "encoder", the results are here:

Name: Fratello_encoder.jpg Views: 8816 Size: 69.7 KB

As you can see, it works.

Have you verified that PortB.5 does go LOW?
Have you verified that you DO get a LOW on RA0,1 and 2 as you rotate the encoder?
Have you checked the datasheet to see if any other peripherals are multiplexed onto the pins you're trying to use? ADC? Comparator?