Every bit of this 2.5min piece, including ideation and creation of all animation and music, was accomplished in 12 of the last 22 hours. The RunwayML AI is insanely powerful, especially if you’re a fan of surrealism. And I finally am so comfortable with the tools in Scaler that that there’s a lot of flow in what’s a pretty complex process. Enjoy this one… it’s a lot more upbeat than my last one (F Lydian doesn’t hurt in that regard!).
3 Likes
That’s really good thanks for sharing, I might look into that, is The Runway ML AI expensive?
Glad you dig it… was an interesting process and I’m just beginning to learn how to wrangle the RunwayML AI. It’s $100/month for unlimited rendering time, which for me seems to be necessary because of all the iteration required to get pleasing results. Here’s a piece I made with it a few days ago, soundtrack just me banging on my hand pan. “Magellanic Cloud” on Vimeo. This piece was much easier to make because it was simply a set of different shots that I cut together, compared to this epic 2.5min seamless single shot.
A friend just asked me to describe how I made this and I’ll copy that description here:
Here’s the basic workflow for this piece, apologies in advance for the run-on sentences.
Started with that portrait of the fly that I made up on the mountain this past Saturday. This was a very chill fly that let me put the phone 1” in front of its face, we spent quite a bit of time hanging out together that afternoon.
Brought the portrait into RunwayML Gen2 as a starting image and hit Generate 4x. (Note that if you give them $100 you can essentially do unlimited generates for one month.). Each generate makes a 4” clip and you can extend each one 4x, for a total of 16” max per clip. Once I get to 4 16” clips I save all of those and scrub through them, looking for a frame that has any sort of cogent visual continuity (the AI really wants to devolve into abstractness when feeding it this sort of material).
Find a frame that feels right, snapshot it and start the process over again, making a new set of 4 16” clip “branches.” Sometimes they are all dead ends so change the random number seed and try again. Rinse and repeat. I have found that word prompts in Gen2 will only confuse the AI if you give it images… it can go off the rails extremely quickly. Best for me has been to use this process to keep nudging the machine towards pleasing visual outcomes, but heck… this is all so new and I’ve only been playing with the thing for 5 days. For this piece that took me ~4-5 hours, with Runway cranking on 4 simultaneous shots the entire time. (4 is the limit they impose on simultaneous renders)
For the audio… once I had a final animation that was brought into Logic Pro, I made a set of 10 markers that defined ranges for the various visual themes of the animation. Then opened up the Scaler composition tool in F Lydian because I just love the qualities of this mode and frequently gravitate towards it due to its major scale quality, but with the raised fourth that creates a tritone from its tonic, making it feel both happy and eerie at the same time (which seemed to make sense for this animation). Once I had that established I just made a bunch of variations of chord structures and then used those within the 10 sections at various tempos (heavier use of Logic’s tempo track here than I’ve ever done before) to build intervals and progressions that felt like they went with those sections. After that I used Scaler’s performance and timing tools to flesh out the performances (I can play a bit of keyboard but nowhere close to what I can program). Brought those performances into midi and tweaked a few things in the piano roll note editor and used one of Omnisphere Keyscape’s beautiful piano patches, with a bunch of effects from the Soundtoys effects rack (super plate, crystalizer, phase mistress and echo boy). Finally did a bunch of sound design with various tools and the diagenic ocean and seagull sounds. This took around 4-5 hours. And voila, done! 
1 Like
Thank you!! Really nice work and on top of that, sharing the process you went through on RunwayML. I’ve been looking into using it for some of my vids. This is really fantastic help! Thanks again for sharing it.
1 Like
I’ve been thinking about doing my own animated shorts, this may be the platform I want. Just retired, though, and $100/mo is a little scary.
It’s not really very much considering how much actual animation would cost in terms of both time and money. That said, this kind of animation is fairly bizarre and not appropriate for a wide range of uses. Yesterday I did a test with some handpan music (my meditation instrument of choice) and got some pretty interesting abstract patterns out of it… this may be my favorite use for Runway. And I probably will only be paying for the service for the first month and then stepping back from it a bit.
I absolutely loved this - I found it mesmerizing! And inspiring. I’m moved to check out Runway. Please do more, and please do post the links.
1 Like
Hey thanks Randall, we’re in the same boat re being mesmerized but this stuff. Runway is extremely bizarre but I have felt compelled to understand the machine a bit more to get a better sense of what the hubbub is all about, which I now do. It’s essentially (and intrinsically) soulless, but with enough wrangling (ie, sending it down many different paths and then finding one that is worth concentrating on) it can be interesting. In that “Panning 4Soul” piece referenced above I added some audio-reactivity to the low D notes so that there are some glows and motion effects that connect the picture to the music a bit and that seems to be an interesting direction in which the AI-generated imagery becomes hybridized with the music in a way that makes it a bit less soulless. I start experimenting with a completely different algorithm today for the next couple of weeks on another video project (a composition about microscopic mites that live in our serpentine rock here) and look forward to posting that before the end of September… the music is at least 70% done.
Really great, Gary - I look forward to more. I’m just getting to the point of quasi-comfort with Scaler, Studio One, and a few other VSTs to work up to pop & jazz compositions. Adding video right now is both daunting and tantalizing.
1 Like
With Runway you just need a huge amount of patience because it only makes 4" chunks, up to 16" long (4 chunks) and then you have to do that screen grab thing I mentioned above. Each chunk takes a few minutes and (with the $95/month plan) you can make 4 at once. So you’re juggling a ton of stuff but also waiting for it to finish, which is like watching paint dry. And since you need to send it on multiple different “expeditions” in order to find one that makes any sort of coherent visual sense, it’s a huge time waster. But all that said, it’s still a lot faster than making animation from scratch! I’ve been in the field of animation and film since 1986 so I’m used to being patient, lol.