Put an elco in series with your speaker to avoid dc current into your speaker.
A potmeter to correct the amplitude for the speaker may also help.