BachDuet: A Deep Learning System for Human-Machine Counterpoint Improvisation

Christodoulos Benetatos, Joseph VanderStel, and Zhiyao Duan

This project is supported by the National Science Foundation under grant No. 1846184, titled " CAREER: Human-Computer Collaborative Music Making".
Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

 

The Proposed System

BachDuet enables a human performer to improvise a duet counterpoint with a computer agent in real time.

The main characteristics of the system are:


Motivation

A system like BachDuet can be used both for education and entertainment purposes. As most user's agreed, interacting with BachDuet, can be an enjoyable method to improve their improvisation skills.


Approach

Model

We quantize music timing into 16th note steps, and at each timestep, the midi LSTM unit t predicts the midi note xt+1 to be play at the next timestep

A key LSTM unit predicts the current key keyt+1 (on every beat), which is used to determine the spell of the midi note (used for score visualization in the GUI)



Graphical User Interface (GUI)

We developed a GUI using the PyQt framework, which serves two primary purposes: a) musical content visualization, and b) system setting control.

Some of its main characteristics include:



Results

Research Questions

We designed experiments in order to answer the 3 following research questions:

  1. Is interactive duet counterpoint improvisation a feasible task for classical musicians?
  2. Do users find BachDuet a good partner for duet counterpoint improvisation?
  3. How does the quality of human-machine duet improvisations from BachDuet compare with that of human-human ones?

User Evaluations

First we recruited 13 musically trained participants and asked them to interact with BachDuet (HM task) as well with another participant (HH task). Then they answered the following questions about the system:


  1. Did you find the GUI easy/intuitive to use?
  2. How accurate were the visualizations compared to your actual input from the MIDI keyboard?
  3. How easy was it to synchronize your playing with the metronome and the machine's output?
  4. Do you think that the algorithm learnt how to play in the style of chorales?
  5. How do you describe the direction of information flow between you and the machine (1 from you to the machine while 10 for the opposite)?
  6. How accurate was the key prediction? (Remember key is different from chord)
  7. How accurate was the prediction of the note spelling?
  8. How did you like the musical result of your interaction with BachDuet?
  9. For the bad musical results, do you feel it was algorithm's fault (1) or yours (10)?
  10. Do you feel that spending more time playing with BachDuet would improve your improvisation skills?
  11. Rate how responsive the system was.
  12. Rate how stimulating/engaging the system was.
  13. Rate how surprising the system was.
  14. What is your overall rating of your interaction with the other person?
  15. What is your overall rating of your interaction with BachDuet?
answers

Subjective Listening Test

For this test, after extracting 40 duets (1-2 minutes each) from the previous HH and HM tasks, we recruited another group of 48 participants (22 from Amazon's mTurk platform, and 26 from a webpage we designed). We asked them to listen to five randomly chosen duets and decide if they correspond to a HH or HM task.

The results of the listening test are similar for both groups of participants (mTurk, webpage), and indicate that the HM generated duets were at least as good as the HH ones, if not better.


turingScoresImage
Confusion matrix
mTurk webpage
true\predicted HH HM HH HM
HH 44% 56% 46.43% 53.57%
HM 47% 53% 48.97% 51.03%

Demo Videos

In all videos, the human performer is the top voice




Code

We provide code for both the original python implementation and the newer javascript implementation.

Play with BachDuet

You can play with BachDuet using computer keyboard, mouse, touch screen, or a MIDI keyboard here!


Publications

Christodoulos Benetatos, Joseph VanderStel and Zhiyao Duan, BachDuet: A Deep Learning System for Human-Machine Counterpoint Improvisation, in Proc. New Interfaces for Musical Expression (NIME), 2020.