In theory, the impedance of your circuit is the impedance of your LM324 output. Few tenth of ohm from what i remind. So it shouldn't be the problem.

What happen if you place a 0.1uF between pin 3 of your LM324 and GND?

Can you monitor the LM324 output with a scope and post a screen-shot of it here?