Context

Project carried out as part of my training in artificial intelligence. The objective was to automatically generate piano music sequences by exploring deep learning approaches applied to musical data.

Description and goals

To test different methods of automatic music generation, particularly through GANs and RNNs, to produce MIDI files that are plausible in terms of rhythm and melody.

Technical Stack

• python Libraries used:
  • PyTorch
  • pretty_midi
  • numpy
  • matplotlib

Implementation

Initial attempts with GANs

We first experimented with two variants of GANs:

• Generating spectrograms from MIDI data,
• Directly attacking the raw audio signal (MP3) via 1D convolutions.

Although promising in theory, these approaches proved inconclusive in our case:

• The generated spectrograms were very noisy,
• The model suffered from underfitting,
• And the generation lacked temporal consistency, which is essential for music.

Final solution: RNN on MIDI data

Due to the technical limitations encountered with GANs, we refocused our approach on Recurrent Neural Networks (RNNs), which are particularly well suited to sequential data such as music.

We used the MAESTRO dataset (classical pieces in MIDI format), which allowed us to work directly on the notes rather than on the audio signal.

Main steps:

• Preprocessing: extraction of pitch, step, and duration attributes for each note, followed by normalization.
• Modeling: an RNN with an LSTM layer, followed by output heads for each parameter.
• Training: with the Adam optimizer and a combined loss function (classification + regression).
• Generation: creation of MIDI sequences note by note, starting from a seed, with "temperature" management to control diversity.

Results and limitations

The results generated are modest; the pieces produced are coherent over short sequences, but quickly show repetitive patterns, inconsistent notes, or unstable rhythms.

This result can be explained by several factors:

• Limited development time (a few weeks),
• Significant hardware constraints: insufficient local GPUs, no access to high-performance servers for long training sessions,
• Small size of the dataset used,
• And a lack of fine-tuning of the model (limited number of epochs, few hyperparameter adjustments).

That said, the project was not intended to result in AI that generates commercial or publishable music, but rather to serve as an experimental exploration. It demonstrates:

• What is possible under limited conditions,
• The sensitivity of architectures to the nature of the data (audio vs. MIDI),
• And the importance of temporal memory (hence the superiority of RNNs over GANs in this context).

Areas for improvement

• Train on a richer dataset (e.g., complete MAESTRO, Lakh MIDI Dataset)
• Add regularization layers to improve musical diversity
• Test more modern architectures such as Transformer (e.g., Music Transformer, MuseNet)
• Integrate harmonic constraints (e.g., tonality, scale, ABA structure, etc.)

Code & demo

The source code is available on this GitHub repository.