I tried the full erase and it didn't change anything.

The config fuse is in the code and I didn't change it in the software. I did another test to where I simply opened the original hex file and did a save as. The original was 2762 bytes and the new file was 2851 bytes. I'm thinking it is something with the software now. I'm going to play around with the settings and see if I can figure out what is causing it.