The oscillator speed must be chosen to meet your overall design objectives. There is no point running at 20 MHz if you are driving slow devices with the PWM output. There, 4 MHz may be a better solution. Likewise if you need specific timing pulses from TMR0 or TMR1 then you must balance the timer bit length with the clock speed and the desired output rate. If you want longest battery life then slower clocks are better.

In general a faster clock lets you get through more code per unit time so faster is better but you must look at the other system requirements.

HTH
Brian