I just found what was bothering me: wrong oscillator fuse setting.

Couldn't see this for hours.....