This might be a read-modify-write issue.
The 18FK4580 has Output Latch registers.
You should try writing to the port via the LATA register instead of the PORTA register.