I'm using two 8580.
I re-listened to the noise closer, and I'd like to give a more precise description of it (sorry for not doing this earlier):
It's more pronounced on the L channel, on the R channel I can barely hear anything. The noise is made of a high pitched hiss and a series of relatively high-pitched crackles. Sounds a bit like a digital version of a scratched record. I don't have the typical ground loop buzz.
The noise also changes a bit depending on the patch, some parameter in the SID is changing the sound of the noise as well...
Could it be a problem with the L Sid? I could try to swap it with another one? I have a spare one somewhere...
Regarding the PSU: I tried to get a good one, regulated, 1000mA...