My guess would be switch bounce. Wait on entry into your interrupt handler for RA4 to return high, or insert a short delay in the beginning of your interrupt handler, and see if that clears up your problem.