What impedance is the speaker ?

I normally use a 0.6W metal film resistor but it shouldnt really be taking any power under normal usage.

Are you sure you dont have some high frequency instability being introduced above the range of hearing ?