Since I don't know what you're doing with your 5v rail, it's impossible to comment. If you're running ADC's and VRef is strapped to Vdd then sure you will have to compensate for that in your design.

Life is a compromise.

It all depends what you're building. If it's a one-off prototype for your own play, then shove a DPDT switch or Relay on the board instead of my Diode solution. Ugly, but no losses there (other than in PCB Real-Estate and Money). If it's a production PCB, then what are you doing with ICSP anyway?... you're just adding unnescessary expense, and if you must, then as I previously said, you will have compensated in your circuit design appropriately.