I managed to solve this one myself. For some reason it works if i put something else in parallel with the solenoid. Im actually using one of the lights for my smoke machines that tell you when you can press the button. Seems to be working fine as long as thats connected.

Does anyone know why this is happening and have any better ways to fix the problem?