The 500ms timeout seems about 10x too long as I'm using 50ms pulses successfully, though on higher voltage SAMs. If anything is slowing down the MIDI communications then it's possible that you are bouncing the SAMs sufficiently that they actually do switch back on for a moment and cause the fluttering. I realize the 500ms timeout is there as a safety precaution and shouldn't be coming into play of course.
Also, the inputs from the SAMs need to be as well debounced as you would do for keys. I assume that's the case. Sorry, I'm not totally up to speed on your hardware but am offering some advice based on similar projects. I like Pete's last advice as well having recently fought with a couple of "spooky" matrix issues myself.