I know it might not be clear from the images but every single screw hole in both PCBs (and the metal connector) are all definitely connected to PCB ground. The ground plane is on the bottom of the PCB and top is 3.3V which is why the screws holes aren't connected to anything on the top of the PCB. They are plated through to the bottom though.

The PSU secondary isn't grounded internally (though I will try that as you suggested). It is still grounded though because it connects to the PCB ground which has the green wire connected to it. It's a longer path but it's still connected to earth.

Is removing the green wire from the PCBs just to see what happens or are you suggesting this as a permanent fix? If I replace that capacitor looking thing with a short then I guess everything would still be grounded but I'd feel better knowing I have an actual earth wire attached in case one of the connectors comes loose.

relying on the psu cover to be screwed on for earthing continuity is not a good look.
Is this referencing the blue capacitor like thing that connects to the case rather than directly to the earth terminal?
I don't think that would be so bad as the case is always supposed to be on and I have separate grounding wires for the PCB grounds.
I could add a wire between the earth terminal and the negative secondary and also an earth wire directly to the case just to be sure.

I'll test everything you've suggested as soon as possible and report back. I know someone who has a better multimeter so I will see if I can try with that.