Scaler slow to load chord sets with 1000's of converted MIDI files

I managed to convert all the MIDI files I have into Scaler format (script set to only convert midi files with chords in them) and then converted them into Ripchord format, for quicker chord browsing. Scaler is an excellent program but having a lot of chord sets really makes the loading of the chord sets slow down to a snail’s pace. I still use scaler in my plugin chain after ripchord, for all the added functionality that it has but really hope that they can address the speed of loading your own chord sets, especially when they run into the 10’s or 100’s of thousands.

From someone who understands computers, this doesn’t make any sense. The code is the same, yet the computer is executing the code at a slower pace…

Please explain what you are doing, as there is something going on in the way you are doing what you are doing. There isn’t enough information for us to reply to your request at the moment.

Please be more specific.

Add 1000’s of Scaler Chord Sets into your Chord Set folder and then see how long it takes to load when clicking to view the next Chord Set. I cannot explain it any other way. I reused the uuid from one Chord Set for the rest of the Chord Sets, since I have a script to partially automate the turning of my midi files into Scaler chord sets, so this may be the reason but there is no chance that I would manually add 1000’s of MIDI files into Scaler, so it was the only solution. It would take months, if not years to do so and it just seems to be a horribly tedious task.

You will know what it does it for. Obviously it has to take time to do it, and in forums I often find similar articles. Certainly speaking, commenting, suggesting new functions … will surely lead to improvements in the plugin itself, but for me, Scaler is useful because I make my chord progressions in it and then use some of its functions; but in order to make or inspire me to make MUSIC. For me in this it is very useful. And yet, to do something that I really like, it takes me a long time to get it done. If I don’t like it, I’m not trying to convince anyone to like it (this is my fundamental maxim)
In those kinds of functions that you think I have never experienced. Every opinion is respectable, but I think that fundamentally, Scaler is for something similar to what I am looking for; I don’t think they made this plugin thinking of something other than musical

Hi @jjfagot. I totally agree with your opinion. I’m not into producing - simply want great chord progressions at a button click, so I can play my VST synths / create arps / use the performance functions in Scaler, etc. as a hobby. MIDI is fun but I prefer to play the chords myself, albeit with a single key per chord. This is what I use Scaler for.

Flooding a system that expects 1:1 (UUID:File) relationship with 1:N relationship is never a good idea. If developers decided to use Universally Unique identifier, there is a reason for that. If I were you, I would generate 1000 UUIDs v4 online and assign them sequentially to each file as a part of your batch convertion process.
Having said that, user files are directly accessible from GUI - this implies that upon Scaler start, a List of Values is created. Something might get cached internally. You might be simply reaching the point where performance drops.

Hi @lelek. Thanks for the info - I’m not a programmer but get what you mean. I assumed that it had something to do with the uuid. Using ripchord as my “chord progression player” for scaler is a workaround that’s working fine for me, although ripchord doesn’t have the ability to place your chord sets into separate subdirectories, like Scaler does. I’ll just save the chord progressions I like into Scaler, to avoid slowing down the loading of the chord sets.

No prob. I’m glad you found a workaround that works for you. Personally, I’m very lazy - I just drag my midi files on Scaler and it gets auto-detected :grin:

Out of idle curiosity, how much CPU, RAM, and/or disk usage is involved during that conversion process?
Maybe that your PC simply is unable to cope with?

Hi @Jason_ZA

Scaler will try to figure what to do with those duplicated UUIDs and from memory will try to reassign a different one on load to avoid clashes with existing content.
This operation reading and modifying files on disk would definitely slow down Scaler enough to be noticeable. We will investigate on our end if we can improve on this, but it is a bit of an edge case.

There is an old post (Scaler 1.2 :rofl:) with some info on the format that is still valid:

Hi Claudio. The conversion process is done before using the chord sets in Scaler, so has no impact at all on resources when using Scaler. I also run the script to remove all duplicate chords from the midi files, so only unique chords are exported.

Thanks @Ed1 - appreciate the feedback.

But it must have an impact on your PC, because it’s the PC that does the process… :wink:

If you have Windows, you can activate Task Manager and look at CPU, RAM and disk writing. I suspect you’ll have some surprise there, as processing many little files can impact a PC as much or even more than processing one multi-gigabyte file

my 2 cents

I run the conversion process during working hours, when I’m not using my personal laptop. It will use resources but since I’m not using my personal laptop at the time, it is of no significance to me (except a slight increase in electricity usage). The script automatically runs through all the subdirectories in my MIDI folder and creates a new cloned folder structure that contains all the files converted into Scaler format. I then run another script to convert the Scaler format files into Ripchord format. The only issue I encounter from time to time is that the script fails due to some issues with certain midi files, so I then have to rename the extension of the problematic midi file (shown in a log file that pops up when an error occurs) and then move all the folders before this folder to another location and then re-run the script (so I don’t unnecessarily re-convert files already converted. It’s a rather seamless process, provided the MIDI files don’t have any errors, which is a rather rare occurrence.

Sounds like you have just hit a memory issue. Scaler comes with what? couple hundred options? Maybe more never counted them :slight_smile:

You then add in your words 1000s more.

So each time you run scaler it has to read all that into …something…be that a temp database, or memory array… something.

Then when you press the NEXT button Scaler has to go… Okay where am I? And where am I going? It then has to increment its UUID(Presuming they are IN numerical order exactly) and then go look for the new one.

This takes time. For any machine. When you go from 100s to 1000s you are actually adding exponentially to the search time needed.

And that is BEFORE you get to the fact that by using the same UUID on your files scaler has had to go thru and assign a NEW Temp UUID to all the duplicates and then keep track of which is which :slight_smile:

Given the sheer volume of files you added this might be something difficult for the devs to fix.

ORRRRR it could be totally easy depending on how they do their sorting, etc. :slight_smile:

But at the end of the day I think you have just hit upon a current limitation :slight_smile: Might be interesting to start removing files until the change time gets back to where it was and see how many you still have added.

1 Like

It is more or less what I was thinking: managing tons of small files can be as hard as managing one only BIG file for a PC, but the poster seems believing it isn’t the case