Do you have any Switch Mode Power Supplies, either 5Volts or other?

The noise might be because of that. If it is on a specific pattern and rate then something else cause this. Either clock, interrupt or as you said USB action.

Critical decoupling may come to rescue. Also earthing correctly also may help.

Ioannis