Maybe your crystal will not run at a voltage lower than 3.6, but the 4550 will operate with an external OSC of 4MHz at 3 volts. I typically run this chip at 3.3 with SD cards.

If you want to save power and run at a lower voltage get a three pin resonator at 4MHz.