I've tried with and without a resistor on the Rx and Tx Pin, there's no difference.

Does it make sense that the serial port seems to be sucking the life out of the circuit...?