Do you have the BOREN configuration bit set? If so, it looks like you'll be in brown out reset if you are under 3.65-4.00 volts.