/ News "Max Order", web-comic on Flow Machines (screen capture). © Fiammetta Ghedini

How Sony Flow Machines’ AI takes itself for the Beatles

Published 14 October 2016 by Elsa Ferreira

Welcome to the future. In less than a month, two titles composed by artificial intelligence have exceeded 1.5 million views on the Web. Will Sony Flow Machines’ musical AI be the first AI-star?

For four years, Sony has been working on an artificial intelligence (AI) project, Flow Machines, a composer of pop, jazz and Brazilian music. Its first titles made public early September, Daddy’s Car, pop inspired from the Beatles and The Ballad of Mr Shadow, in the style of the American songwriters Irving Berlin or Duke Ellington, were listened to by more than 1.5 million internet users. Is this the beginning of a new musical vein?

“Daddy’s Car”, Flow Machines’ most popular song:

The program was developed within Sony CSL Paris, research laboratory celebrating its 20th anniversary that works on a bunch of subjects, from automated permaculture to computational linguistics. Its equivalent in Tokyo is working on prostheses (that we were able to see at the very first Cybathlon in Zurich), energy or augmented reality.

For Flow Machines, François Pachet, researcher at the Sony Computer Science Laboratory in Paris in charge of project development, received a five year grant from the European research council and is working with around ten people with various skills: sound engineers, musicians, programmers or physicians.

Dissection of a song

In order to explain how his program works, François Pachet started by dissecting the functioning of a song into four stages:
–First stage: a lead sheet or “symbolic information” in cognitive science;
–Second stage: the realization, the orchestration or the arrangement. i.e. “ how you put sound on musical notes”, explains the researcher.
–Third stage: production, i.e. “the special effects you apply to sound”, compression, equalization, or still panning;
–Fourth stage: performance.

13,000 lead sheets to feed the machine

For the first stage, engineers integrated lead sheets of jazz, pop and Brazilian music in the database—13,000 in total—as well as all the lead sheets written by the Brazilian Almir Chediak, “the God of lead sheets” according to Pachet. “Many lead sheets are no good. Chediak is one of the first to have made faithful transcriptions of Brazilian music stars.”

Why limit yourself to pop, jazz and Brazilian music? The catalogue was at first chosen for its “closed” characteristic: the style of dead composers, such as Miles Davis for example, is easier to model. And these music genres focus on melody or harmony. “With lounge music, hard rock or rap, the notion of lead sheet is much less important because there is no harmony, or very little”, explains François Pachet. With no value judgment, he insists.

The machine then launches itself in statistical representation work. “At the beginning, we gave a style, i.e. a set of songs we selected ourselves—in machine learning, we call it the training set. The machine will analyze them and try to see what is recurrent. After such-and-such note, such-and-such chord, there is rather such-and-such note and such-and-such chord with such-and-such probability. Its is done in an intelligent manner and it builds a statistical object from which one will be able to generate other pieces that will have the statistical properties observed in the corpus.”

Bossa nova, Mozart-style

For the second stage, the programs Flowcomposer and Rechord, also developed at the Sony CSL, come into play. They allow you to create accompaniments in a given style. “We recorded human beings playing The Girl from Ipanema, says Pachet. The Rechords system takes this accompaniment and cuts it into pieces, analyzes it and synthetizes an accompaniment for another lead sheet that will not have the same chords or the same rhythm.” And compose a piece from bossa nova in Mozart’s style for example…

How Flowcomposer works, Sony CSL Paris:

The mixing is provided by a system developed by Sony, Automix, that should soon be commercialized. Even though the program does not know how to make esthetic choices or carry out “more creative tasks”, “it is capable of doing a premix, doing lots of cleaning, saving tracks, equalization” adds Pachet.

Finally, for performance, researchers set up the Reflexive Looper, that “allows you to play a lead sheet by doing in real time more or less what an orchestrator would do: you play a little guitar, the system automatically understands what we have played and plays the piece of guitar that is just right in the future of the song.” This program, more intended for live performances, is not included in Flow Machines.

The sound one hears in the first titles of Flow Machines is therefore a “true” sound of a recorded instrument that has been incorporated in the base and chosen by the machine.

How the Reflexive Looper works, Sony CSL Paris:

The place of the human

Flow Machines could be totally autonomous, but François Pachet says he is seeking quality. “A good song is rare”, he repeats. “A song tells a story: a beginning, a middle and an end. Today AI techniques are not very good to produce structure.” A dialogue is established between the artist and the machine, explains the researcher. The lead sheets appear on the screen and the composer (human) can edit them. “You can say for example: “I like the beginning but not the end, give me another solution.” You can hold a dialogue until you get something that you like.”

Benoit Carré, the composer who worked on the first released pieces, has been following the researcher’s experimentations for a long time. He tells us (via email) that he was “seized” by the Flow Machines experience. “An internal dialogue of a whole new genre” was established between the machine and himself.

“The machine pushes you to your limits. The melodies generated confront you to a choice. Your instinct or your intuition are highly called upon. The machine helped me to create a song for it – weird sentence but not false! There are no limits to your “creativity”, it’s up to you to give it your choices, to establish benchmarks. Once the framework is defined, if it is sufficiently coherent (if you make the right choice in your selection of source lead sheets), the melodies generated can be very inspiring.”

Benoit Carré, Flow Machines composer

Half a century of musical artificial intelligence

Even though both pieces released by Sony were presented as the first pieces composed by artificial intelligence, researchers have been taking an interest in the subject for several decades. The first computer generated creation, Suite Illiac, composition for a string quartet, goes back to 1957. In the 1990s, the researcher and American composer David Cope also released several albums generated by a program capable of emulating classic composers.

Last June, it was Google that released a 90 second ballad composed by its program Magenta. A naïve and simple melody. A “joke” reckons François Pachet. “Google made an idiotic communication operation. Because, besides, they have very good technologies.”

The composition of Magenta:

“The term ‘first’ stirred controversy, admits François Pachet. But it’s truly the first time we have a song with a professional level. It’s very hard to go from the technical demo for colleagues to something that people listen to. There is a wall between the two.” So the first (or nearly) to exit the lab, except for Pierre Barbaud, “a rather eccentric guy” who until his death in 1990, composed television jingles with AI, the researcher recalls.

Pierre Barbaud’s “French Gagaku”, 1971:

For the time being, the technology of Flow Machines is not accessible to the general public. “It is very expensive to make developing software public, we are a small lab”, justifies the researcher. Sony on the other hand envisages to release three albums. The first will be Mr Shadow, from Flow Machines and Benoit Carré. “It’s a bit like exploring one’s dark side by setting aside the composer’s ego’” says Benoit Carré. Then a collaborative album with Alb, Barbara Carlotti and “others better known whom I cannot mention”, teases Pachet. Finally, the “next Beatles album” will be released. “Not imitative but as if the Beatles had continued up to today. Because they were always ahead of innovation”, explains the researcher. In the meantime, Flow Machines is also the subject of a web-comic, Max Order, for which the machine composed the soundtrack. A star is born…