your not trying to run chip at less than 5v ? decouple caps etc [i'm using none]
reset is pulled high ?
enc and sw pins are pulled high with suitable r

i cannot get mine to fail no matter hard i try , no caps, pwr off pk4 , dupont leads on a breadboard [you can't get worse conditions]
even using wpu is ok

i will go out on a limb and say
code cause 0%, isr cause 0% . other 100%

also when breadboarding all unused pins set as i/p is a good thing