Neural Music
Svitlana Leus, Daniel Kiss, Edoardo Abate
Introduction
Motivated by our lack of musical talent and programming skills, we thought it would be a great idea to make Artificial Intelligence compose some music, and make it accessible via a web app. Here is how we did it.
Web Development
Our goal was to create a minimalistic but modern looking web application. We considered different options of the website structure, splitting the logic to different pages, and linking them together. But eventually, we ended up creating a Single Page Application.
Our technology stack: HTML, CSS, JavaScript, Bootstrap 4, JQuery.
The first challenging step was to create the website design or at least to come up with a final decision of the key design components. Not being natural designers in our hearts, but at the same time having high expectations, we were switching from one idea to another at least 30 times.
Our main source of inspiration was dribbble.com — the platform where thousands of designers are sharing their works. It was easy to catch the idea of the main trends of web design in 2020. Another lifesaver was codepen.io. Here we could find some great, and free, examples of components that we wanted to implement, but had no idea how to do it.
The interaction with our model is very easy and consists of 3 steps. Let’s take a closer look at each component that was required.
Artist selector
As we expect to get only one option selected, the obvious decision was to use radio buttons. Of course, we did our best to prettify them — we changed the regular look and added some animation. After making their decision users are supposed to press the “Generate” button to proceed further.
When user press this button two things will happen in the background:
- An asynchronus AJAX call will request the model to generate a melody, considering user’s choice of artist.
- The “waiting” animation element will appear below. This animation was borrowed from loading.io, which provides many animaiton examples.
Once the model has generated the MIDI file, the AJAX request receives a response with the path to the generated file. All is left is to load the generated file using the JavaScript library MIDIjs, another awesome resource we would like to pay a tribute to.
Artificial Intelligence
Art generation is an interesting challenge and music has been object of study of several researches. A great inspiration for our project was drawn by the work of Briot, Hadjeres and Pachet (2019), who have shared their analysis on more than 200 works that used Deep Learning to generate musical content. In general, different data types (raw audio files, MIDI, text) and various architectures (RNNs, to autoencoders and GANs) have been used with different results.
It is easy to get overwhelmed by what is possible to do in this context, so we opted for a simple approach. As a form of representation for music, we relied on MIDI files. They are commonly used interfaces for interoperability between different electronic musical instruments, software and devices. The Internet provides an abundance of them and python libraries like Music21 make working with them relatively easy. Music21 was used to build a vocabulary of notes and chords, which was later mapped into categorical data. In training the network, an arbitrary sequence length was used as an input with the aim to predict the following note or cord from the vocabulary (the target). The model was setup relying on Keras, an open-source neural-network library written in Python, and it features several layers (LSTM, dropout, dense), and categorical cross entropy to calculate the loss. For the curious ones, the model can be found in the generate.py file inside the GitHub Repository accompanying this project.
Figure 3 Output after training for 5 epochs.
The model was trained for about 200 epochs for each featured artist, for a total of about 50 hours. This was only possible by using Google Colab, since otherwise our computers would have needed up to 800 hours to perform the needed computation (no thanks!).
Each artist has its own vocabulary and weights, which are used to generate the notes and chords sequences. As a starting point for the generation, a random element from the artist’s vocabulary is used. Finally, Music21 is used once again to rebuild the MIDI file from the generated sequence. Our generation code is wrapped inside a function which takes an artist as an input and returns a path to the generated file.
What we take away
The learning curve was quite steep and the project was demanding, but our commitment eventually paid off. We are happy to have brought this minimal web app to life and our respective TechLabs tracks were key in giving us the tools to make it happen. In wrapping this report up, we would like to say thanks to all the people behind TechLabs who spent their time and energy to make our journey possible. We would also like to thank our mentor Kristoffer Clausen, for giving us valuable feedback along the way. Finally, to those who have shared, and keep doing so, their code and libraries freely on the internet, thank you, you are awesome!
There is so, so many things we wish we had the time and knowledge to develop, but we had to live with some limitations. Who knows, perhaps some future TechLabs Project can pick up from where we left!
Team members LinkedIn profiles:
- Svitlana Leus, Web Development — Front End Track
- Daniel Kiss, Web Development — Back End Track
- Edoardo Abate, Artificial Intelligence
