Hi,
Change the code so that you interrupt on one edge of one channel - only. (The rising edge of channel A for example).
In the ISR, poll "the other" channel, if it's low you're moving in one direction, if it's high you're moving in the other.
Verify that it works at "any" speed you may turn the encoder.

/Henrik.