As Richard says, you only need one interrupt. On interupt edge, read the inputB from the encoder . If it's hi then being turned c/w, if lo then being turned ac/w ( or the other way round) Suck it and see.

Also good advice from Richard re caps on both inputs to ground.

You can almost discount switch bounce, int will ignore following spikes and inputB switch will have settled. Only problem is if interupt input is still bouncing after you reenable ints. I added a test for no change on inputB.

George