I thought of one more thing, if your encoder wires are really long the capacitance of the cable may be rounding off the tops of the signal and the PIC just might not like that, in which case and amplifier may be required.
Good Luck
JS