rotary encoders woes are a common issue have a look at this thread


http://www.picbasic.co.uk/forum/showthread.php?t=13177

secondly if your encoder is mechanical type contact bounce is an issue that must be dealt with