That helps a ton.
so I don't know if my approach is flawed, I have something wrong in the code, or there is some hardware problem.
It may be a little of each. But first lets attack the hardware. I don't think you need the "R" signal. you can get the counts from the A or B. Now, the reason I want to drop it, besides making the code a little simpler, is I fear you may have some overlap in the fields. This could be why you get the wierd stuff with rising or falling edge trigger. Do you have access to a scope to actually watch B and R to see in B dies before R rises?