Investigation of Scaler MIDI and audio detection - first conclusions

[yorkeman]

Part IV - Conclusions and Thoughts

Conclusions :

1 Scaler is very good at classifying those chords it can reliably detect. Every instance I checked where there was the appropriate number of simultaneous notes without extraneous non chord notes agreed with the definitions in my (good) guitar scales and chords utility.

2 Scaler (both audio and MIDI) worked less well in particular cases, such as where pads were played legato in adjacent bars. This is a normal circumstance where a user wants to repeat the attack part of pad rhythmically. The logic appears to detect changes to the detected chord, but disregards bar boundaries, even though there is a ;note on’ event. This will tend to affect ambient and cinematic passages most.

3 Scaler also had difficulties in detection where there were insufficient simultaneous (and ambiguous) notes. However, a chord is three or more notes played together, so this behaviour isn’t a bug per se, as it is still doling what it says on the tin.

4 Although technically excellent, the chord detection in one sense was too good, in that the human player would probably not have used the detected chords, and used something simpler as a bland backing.
In one example the tonality and progression pointed to minor, but the missing third (which was sort of implied) and a flat 7 led to a mixolydian mode being declared, where by ear the player probably wouldn’t have selected this.

5 BIAB detected some pieces well, but probably because it seems to use just triads and sevenths, tying up with point 4 above. It doesn’t have the legato problem and correctly identifies bar boundaries. Its lack of rigidity is paradoxically a virtue in some cases, and hence seems to detect in certain cases things better in the overall context of a sequence. However, it is simply no match for Scaler in terms of scope and accuracy for complex MIDI chord detection.

6 On the audio front, deCoda is rudimentary in terms of detection (casting detected chords as triads), but can work with complex polyphonic material, and is good at finding the underlying key. I felt this offered an opportunity to use this in conjunction with Scaler (see ‘Thoughts’).

Thoughts:

1 The detection quality of Scaler can be improved with minor changes to workflow. So, with (for example) a piano MIDI piece with both melody and chords, spending 10 seconds to chop the right hand part of the MIDI resulted in more accurate and relevant detection. Similarly an audio piece can be quickly filtered with a band pass to carve out pads or other chordal element to greatly improve detection performance.
Thus considering what pre-edits might be done on both MIDI and audio may improve overall success rates.

2 Section B is a secret weapon here. So for example, with a complex polyphonic piece, detecting the key (and changes) with deCoda (and a few hints on the possible chords) and then listening to the piece and selecting chords from the harmonised scale chords of the deCoda detected key, quickly allows the human detection of the progression.

Comments welcome, and if anybody reads this, I will throw in a few other (self-evident?) actions which help detection.

[Ed1]

thanks for this in-depth investigation @yorkeman

It is pretty much aligned with our internal testing and other feedback we have received.
Chord naming and detection both remain a technical challenge as they rely on many different parts with incomplete information.

The perfect detection would need to know the scale of the piece you are getting the chord from in order to name it accurately. Even with the scale known, in some cases you would need to know the previous and/or following chords or the intent of the player to remove any remaining ambiguity.

Knowing this, we developed those features to be useful to most and in most cases. Trying to give you an answer that helps you find related content, to quickly get an idea and start from there.

Obviously the more accurate the guidance the better and we will continue to improve on this front in future iterations.

Thank you again for the detailed write-up, I know too well how interesting it can be to dissect and benchmark Scaler, and it makes for really nice forum posts for us all to enjoy. Just remember to take some time to make music as well, you deserve it

[yorkeman]

Just to be clear, I am really impressed by the accuracy of Scaler’s detection of chords. I had deliberately picked pieces (or edited the input) to create tough pieces to interpret, including ambiguous ‘implied’ tonality, which was hard.

One reason for doing this exercise was to explore simple pre-processing with both audio and midi to make it easier for Scaler to pick up the chords, and it confirmed that quick, simple editing could make a huge difference in ‘noisy’ environments where the input was peppered with melodic notes.

In any event, once the key/ mode was established, the strengths of Section B then shone, as it was easy to listen to the piece and just click on the harmonised chords to pick out the progression.

So, if I were to venture any suggestions they would be

1 To have an option which allowed ‘restrict to triads / restrict to 7ths / no restrictions’, which the user could try and then pick the best (this is handy where one is just looking for chords for simple backing pads, rather than some more complex guitar or piano infill to solos etc.)

2 Having some means to detect bar change to deal with the apparent anomaly arising with legato pads, where two repeated bars are identified as one.

For me this exercise confirms Scaler as an amazing software product.

[Bernd]

@yorkeman

Just to be clear, I am really impressed by the accuracy of Scaler’s detection of chords. I had deliberately picked pieces (or edited the input) to create tough pieces to interpret, including ambiguous ‘implied’ tonality, which was hard.

In SW/IT engineering circles this is called “stress testing”