if you look at the expected b5,b6 pin values at each interrupt
for neg edge trigger only

ints occur on orange lines
Name: re.jpg Views: 1751 Size: 26.5 KB
under ideal conditions you get for say cw rotation
01,00,01,00,01............
for ccw
00,10,00,10,00............





so for cw


old = 01000000 new 00000000 or old = 00000000 new 01000000
new.6 ^ old.5 =1 new.6 ^ old.5 = 0




so for ccw


old = 00100000 new 00000000 or old = 00000000 new 00100000
new.6 ^ old.5 = 0 new.6 ^ old.5 = 1

i don't see your logic decoding direction at all successfully