well without any real details, my first guess is you are overworking your regulator. I am guessing that reg is only good for ~40mA with no heatsink. with heatsink, ~100mA. thats 5 to 10 led's at once with no reguard for other circuit requirments. Another guess is 6V (best case) is not enough for the reg to provide good 5V. so maybe browning out?

Bottom line is more details will be needed I think for any real help to be had.