READ-MODIFY-WRITE (R-M-W).
Especially at 64Mhz.

Use LATB instead of PORTB