Just looking at the schematic again, mrtripoli has already implemented the changes I was talking about.
I mistook the ICSP connector for an EEPROM!

Looks like I can adopt this schematic for the final package
Many thanks mrtripoli.