So after years one planning to do this, I finally got around to it. Here are the results:
Simple TL074 based mixer with stereo pots for each channel that fits to the existing holes. I've got a handful of boards if anyone would like one and then I'll post a OSHpark link so anyone can get them any time.
Boards:
https://oshpark.com/projects/iEI4Yrhq
BOM
https://docs.google.com/spreadsheets/d/1w-VEWmPBQucrhK-vZdvH8pU_xwDvS5GS1VtrLofj800/edit?usp=sharing