Hi,
You can try to identify the bad transitions and use a custom detent-mode. Therefore, instead of MIOS_ENC_MODE_DETENTEDxx, use 0x01, 0x02, 0x04 and 0x08 to test increments, and 0x10, 0x20, 0x40 and 0x80 for decrements. Try them one after another in normal speed-mode and note, which show jitter and which are jitter-free for a full revolution of the encoder.
to get a behaviour like DETENTED2, you need one jitter-free increment and one decrement. For DETENTED (resp. DETENTED1) you need two jitter-free test-modes for increment and two for decrement. Then you can sum up the working test-modes to a custom mode and use that for your application. remember to add the modes in hex!
Just an example: let's assume, the following test-modes are jitter-free: 0x02, 0x08, 0x10, 0x40, 0x80.
to get a custom DETENTED2-like mode, you could use 0x02 for increments and 0x10 for decrements, summing up to 0x12 as your custom-mode.
to get a DETENTED1-like mode, you could use 0x02, 0x08, 0x10 and 0x80, resulting in 0x9A as custom-mode.
I hope you can find a working enc-mode that way.
best regards,
Alex