I have learned a lot, working on this problem, but the problem was that I had all my configuration fuses enabled (except protection). Thanks to ?wumpus (I forget his handle), I disabled low voltage programming and all is well now.