BachDuet: A Deep Learning System for Human-Machine Counterpoint Improvisation

Christodoulos Benetatos, Joseph VanderStel, and Zhiyao Duan

This project is supported by the National Science Foundation under grant No. 1846184, titled " CAREER: Human-Computer Collaborative Music Making".
Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

The Proposed System

BachDuet enables a human performer to improvise a duet counterpoint with a computer agent in real time.

The main characteristics of the system are:

Input: Human musician’s monophonic performance on a MIDI keyboard
Ouput: The machine’s monophonic performance in real time generated by an RNN.
Adjustable steady tempo (metronome) to ensure human-machine synchronization
A GUI to visualize the generated music content and to facilitate this human-machine interaction

Motivation

Today, there is a lack of improvisation culture in classical music performance and education
Classical musicians either are not trained to improvise, or cannot find other people to improvise with
Current interactive music systems are not designed with classical music in mind, and they mostly support the "call & response" interaction mode

A system like BachDuet can be used both for education and entertainment purposes. As most user's agreed, interacting with BachDuet, can be an enjoyable method to improve their improvisation skills.

Approach

Model

We quantize music timing into 16th note steps, and at each timestep, the midi LSTM unit t predicts the midi note x_t+1 to be play at the next timestep

A key LSTM unit predicts the current key key_t+1 (on every beat), which is used to determine the spell of the midi note (used for score visualization in the GUI)

Graphical User Interface (GUI)

We developed a GUI using the PyQt framework, which serves two primary purposes: a) musical content visualization, and b) system setting control.

Some of its main characteristics include:

Real time piano-roll and score visualization
Support for any midi device as input
Saving the interaction to music xml

Results

Research Questions

We designed experiments in order to answer the 3 following research questions:

Is interactive duet counterpoint improvisation a feasible task for classical musicians?
Do users find BachDuet a good partner for duet counterpoint improvisation?
How does the quality of human-machine duet improvisations from BachDuet compare with that of human-human ones?

User Evaluations

First we recruited 13 musically trained participants and asked them to interact with BachDuet (HM task) as well with another participant (HH task). Then they answered the following questions about the system:

Did you find the GUI easy/intuitive to use?
How accurate were the visualizations compared to your actual input from the MIDI keyboard?
How easy was it to synchronize your playing with the metronome and the machine's output?
Do you think that the algorithm learnt how to play in the style of chorales?
How do you describe the direction of information flow between you and the machine (1 from you to the machine while 10 for the opposite)?
How accurate was the key prediction? (Remember key is different from chord)
How accurate was the prediction of the note spelling?
How did you like the musical result of your interaction with BachDuet?
For the bad musical results, do you feel it was algorithm's fault (1) or yours (10)?
Do you feel that spending more time playing with BachDuet would improve your improvisation skills?
Rate how responsive the system was.
Rate how stimulating/engaging the system was.
Rate how surprising the system was.
What is your overall rating of your interaction with the other person?
What is your overall rating of your interaction with BachDuet?

Subjective Listening Test

For this test, after extracting 40 duets (1-2 minutes each) from the previous HH and HM tasks, we recruited another group of 48 participants (22 from Amazon's mTurk platform, and 26 from a webpage we designed). We asked them to listen to five randomly chosen duets and decide if they correspond to a HH or HM task.

The results of the listening test are similar for both groups of participants (mTurk, webpage), and indicate that the HM generated duets were at least as good as the HH ones, if not better.

Confusion matrix
	mTurk		webpage
true\predicted	HH	HM	HH	HM
HH	44%	56%	46.43%	53.57%
HM	47%	53%	48.97%	51.03%

Demo Videos

In all videos, the human performer is the top voice

Code

We provide code for both the original python implementation and the newer javascript implementation.

Play with BachDuet

You can play with BachDuet using computer keyboard, mouse, touch screen, or a MIDI keyboard here!

Publications

Christodoulos Benetatos, Joseph VanderStel and Zhiyao Duan, BachDuet: A Deep Learning System for Human-Machine Counterpoint Improvisation, in Proc. New Interfaces for Musical Expression (NIME), 2020.