In theory it should work... Perhaps the interrupt latency is just too large to be able to catch the correct state of the 'other' channel once you get to the ISR. One way to test that is to try it with the Grayhill encoder, then it should count every 4th click. If it does then it's a timing problem, if it doesn't it's something else.

Nice demonstration of conditional compilation!

/Henrik.