My practice is what Richard noted. Always have either external pull ups or use the port b with the internal pu enabled and active low switch. No other circuit needed and have the Icsp programmer there just fine.

Ioannis