Magenta

Are you familiar with Magenta? [0] It's not real-time, but definitely an important area of research. Also check out Dan Tepfer [1] who is doing amazing work with an algorithmic approach to reactive live performances, with great call and response tactics. I myself am slowly prototyping a fully artificial AI band which can be orchestrated using very high-level musical ideas and a big helping of intelligent randomization and algorithms based on music theory. I've been prototyping in Andrew Sorensen's Extempore [2] and have laid much of the groundwork like melody/harmony/rhythm generation as well as basic modulation of these elements to create longer musical structures which utilize motifs in multiple ways Currently it is a matter of shedding pre-computed tables of "nice" sounding progressions or purely random progressions, and creating a more fundamental approach which can derive the appropriate progressions from the given user parameters. I am also expanding the program's ability to generate aesthetically pleasing and unique singular motifs which drive these algorithmic compositions. I have a band leader / conductor module which provides cues and other synchronized data, which even without advanced motif generation and modulation still allows for things not currently possible in any non-code musical production software such as global dyamics, timing changes, progression sharing, (eventually) directing impromptu solos, etc. Reach out via email if you'd like to discuss…

SEEKING INVESTMENT | Greater Seattle Area | ML, MUSIC Project or startup: Music streaming service with AI-generated music and lyrics Seeking for: $200,000 investment for up to 20% stake Spending on: - $180,000 for 1y salary for R&D - $20,000 for hardware purchase/rent for model training Team: just myself, software engineer with ML background Business model: streaming subscription Some preliminary work: Google showed neural networks can be used to generate music in MIDI format ( https://magenta.tensorflow.org/ ), own project shows GPT-2 can generate song lyrics ( http://billion.dev.losttech.software:2095/ , based on the initial small 117M GPT-2 release ). MIDI format supports lyrics tracks, so the two can be combined, producing karaoke-like experience. There are also singing voice synthesizers on the market, which could be used to automatically generate song recordings. Alternatively, real performers could record tracks.

There has since been follow-up work to extend Onsets & Frames to multi-instrument music: https://magenta.tensorflow.org/transcription-with-transformers

You're right to be wary, I re-read your use case and hand tracking alone won't cut it. However I still think that hand tracking _combined_ with audio input can be precise enough (above 99%). Here is an example of raw audio to midi: https://magenta.tensorflow.org/onsets-frames Regardless of the feasibility, it won't be as reliable as direct midi input, that's for sure! I think your goal of getting things working for a simpler case first is right, and wish you best luck :)

The GPT-2 based Musenet music generator is already interesting but far from perfect. You can try it in the middle of this article: https://openai.com/blog/musenet/ (you can even upload custom prompts in the advanced mode) Would be interesting to see it with the updated GPT-3. There is also AIVA with more production ready results: https://www.youtube.com/watch?v=gzGkC_o9hXI&list=PLv7BOfa4CxsHAMHQ… Not sure how it works, but it has better results maybe because it's using more predefined components and less AI so it's also less "creative". More AI music projects here: https://magenta.tensorflow.org/

Thanks for your resource. Never heard of Dan Tepfer or Extempore - such a great way of imagining music! What I was planning was something simpler - much like generating sounds from a written score, but like live classical performances, the generated sound reacts to the cues of the player. I'm not exactly familiar with Magenta, but the thing I'm currently trying to implement (at a very early stage) is Deepmind's Wave2Midi2Wave which is part of the dataset released with Magenta [1]. I'm not aware if they'd released any code as well. [1] - https://magenta.tensorflow.org/maestro-wave2midi2wave

> After spending the better part of a century on that problem, we have made essentially zero progress. I dunno, I heard music composed with neural nets that is above what the average human could achieve [1]. Not on par with the greatest composers, but over average human level. In the same line of thought, I have seen models do symbolic math better than automatic solvers, generate paintings better than average humans could paint, even translate better than average second language learners. I would rate current level in AI at 50% of human intelligence on average, and most of that was accomplished in the recent years. [1] Google Music Transformer - https://magenta.tensorflow.org/music-transformer

There’s a pretty cool library from Googles Magenta team that does piano transcription pretty well. https://magenta.tensorflow.org/onsets-frames They say it’s only really good for piano, but I definitely use it for all kinds of samples. Great for inspiration

Well if by music that includes music production: Check out Google Magenta for machine learning applied to music: https://magenta.tensorflow.org/ As for industry: probably more ways to sample sounds on the fly into new instrument presets, new ways to organize music (both live and arranged), better ways to collaborate, and better ways to self-market.

That is most likely because Magenta now is a Tensorflow project, talk about communication inside the same company! https://magenta.tensorflow.org/

Coming soon [0] [1]! [0]: http://magenta.tensorflow.org/ [1]: https://github.com/tensorflow/magenta

PLL is an interesting option, never thought of that, that is worth pursuing, thank you. And like the other comment already wrote, state of the art. The project is automatic music transcription, the 'gold standard' that I've found so far is Magenta based 'Onsets and Frames' (https://magenta.tensorflow.org/onsets-frames ), which scores pretty good on a large benchmark. I'm still a ways away from being their score but there is real chance that it will happen. The tensorflow based software is super slow, many orders of magnitude slower than realtime would require and I have a bunch of applications in mind where slower than realtime is not an option. Here are the two side-by-side across the same testset (first mine): // di 24 dec 2019 16:49:29 CET overall total 17482 correct 12604 fp(fantasized) 2723 fn(missed) 4878 f_measure 76.833 precision 82.234 recall 72.097 accuracy 62.381 vs (Magenta O&F): // wo 25 dec 2019 20:09:51 CET overall total 17482 correct 13865 fp(fantasized) 2908 fn(missed) 3617 f_measure 80.952 precision 82.663 recall 79.31 accuracy 67.999 Clearly the O&F software has the edge in detection but it is close enough that I have some hope of beating that score, I'm already way ahead of what they list as the 'previous state of the art'.

TL;DR: If drumbot.glitch.me doesn't work for you, try running it locally [2]. It's surprisingly effortless! First attempt at trying this out: 502 Bad Gateway. Second attempt: I thought I didn't want to mess with Node, so I downloaded Magenta Studio [0] instead. When trying to load the Max patch, my potato promptly lost its will to Live. (Ba-dum tss [1].) Maybe loading a 900MB .amxd when you only have 1GB of free RAM isn't a great idea? Anyway, I'm not keen on risking another reboot today, so I'm putting this off for now. Third attempt: Localhosting DrumBot [2]. Five minutes ago, I didn't have Node installed (I didn't even recall ever touching Node directly at all), but now (npm install … added 247 packages from 284 contributors … found 4 moderate severity vulnerabilities … npm start) the tensors are flowing already! Whoa, Node is so easy, any idiot can use it — even musicians ;) I peck in a simple melody: Fly me to the moon Let me play Fly me to Oops, this started looping at an unexpected point. I guess I should have stuck to a 4/4 time signature? DrumBot is equally confused and lethargically bangs out wonky shit like it's attempting to cure carbon monoxide poisoning with ketamine. Meet my new band: Tourette's Horse Tranquilizer. I start over, this time with a proper MIDI keyboard plugged in. Initially this fails: MIDI Off messages trigger notes in the same way as MIDI On messages. This is easily fixed by routing the MIDI through a DAW. (Why? No idea.) I play a…

Glad to see people posting links to music modeling projects - it's a huge space, and a ton of fun to actively work on or just mess with if you like music and code. Even simple markov models can get someplace with a bit of tweaking [0][1]. If you like this, you might like some more recent work from Anna huang et. al. at Magenta, using a modified Transformer network [2][3]. Some other relevant reading includes DeepBach [4][5], and outside the domain of Bach alone, the venerable folkRNN [6]. There's a huge chain of work I posted a while back here [7], and Kyle McDonald wrote a really great review piece here [8]. Modeling music in latent spaces has also seen a number of recent publications [9][10][11], and having smooth latent spaces for generation seems (to me) to be a very natural interface for manipulating things, which has a natural analogue in "real" composition, cool video at [12]. There have also been some really fun online demos for interacting with these types of systems [13] There's a lot of interesting work going on in music modeling, hopefully leading to new and interesting musical tools and toys. Interactivity (often via clever conditioning) can really change the way you approach these types of problems [14]. [0] https://github.com/kastnerkyle/magenta/tree/my_cleanup_branch/exp/mark… [1] https://www.youtube.com/watch?v=Ay8mHW0-NW8&list=PLRMa_gJ8vx8mC9Vv… [2] https://twitter.com/kastnerkyle/status/1018617315196309504 [3]…

Essentially, the leading way to do automatic music transcription is to train a neural network on supervised data, i.e., paired audio-MIDI data. In the case of piano recordings, there is a very good dataset for this task which was released by Google in 2018: https://magenta.tensorflow.org/datasets/maestro Most current research involves refining deep learning based approaches to this task. When I worked on this problem earlier this year, I was interested in adding robustness to these models by training a sort of musical awareness into them. You can see a good example of it in this tweet: https://x.com/loubbrad/status/1794747652191777049

You can actually use thousands of oscillators, which is what Magenta did with their differentiable DSP (DDSP) approach to modeling acoustic instruments including singing: https://magenta.tensorflow.org/ddsp-vst https://magenta.tensorflow.org/ddsp I doesn't sound realistic but it is expressive in an unusual way.

Google has actually released Magenta [0] which is a plugin within Ableton that can do just that. It's pretty cool, it generates interesting melodic content at the click of a button. [0] https://magenta.tensorflow.org/studio

Not yet though. In the future, for sure. My point is that Napster-to-SD is not a fair comparison. Napster didn't eliminate or replace human artists, while SD certainly did. Therefore, it's not fair to assume the government can regulate themselves out of this one like they did with Napster. Because even though Napster enabled widespread piracy, the human musicians still had leverage in that they were needed to create new music. Obviously, if a good generative model comes out, the music industry will be in a similar boat to the art industry right now. Google was working on it in 2017 (https://magenta.tensorflow.org/performance-rnn) but I don't know if they've made any progress since.

For context, here's some recent computer music, both MIDI and direct audio generation: https://magenta.tensorflow.org/perceiver-ar Here is an older project that generated audio from lyrics: https://openai.com/blog/jukebox/ Music has two key difficulties: the long, linear context makes it hard for most models to generate coherent works and reason about long-term higher level structure, and the global structure being learned is much more function of human preference, rather than of physical constraints. That said, I don't think the gap is all that sizeable. Jukebox is easily comparable in quality to the image synthesis from arbitrary text of its time. Perceiver AR is not as impressive as DALL-E 2, sure, but DALL-E 2 is new, and it's plenty of evidence that music synthesis will continue to work.

The crux of the DDPS architecture seems to be allowing a model to train the many parameters of the sound synthesizer. You would never be able to create a truly realistic piano from a software synth by merely tuning the knobs because there are too many near-imperceptible minor fluctuations in the wave form, like the way the vibrations in aggregate vibrate the wooden piano housing. Training on real life examples, in this case an impressively short clip, gives the model enough data to train these near-imperceptible variations into the synth parameters. https://magenta.tensorflow.org/ddsp

For those interested, there's a piano-centered dataset[0] [0]https://magenta.tensorflow.org/datasets/maestro

I am not an ML expert so not sure of the details on how it works, but I believe Google’s DDSP work is similar to 1: https://magenta.tensorflow.org/ddsp As I understand it, their system can take an audio input and “learn” how to control the parameters of a synth to reproduce that sound. I had the pleasure of seeing a demo of it from someone who worked on it and thought it was impressive! Synths have existed for a while which can use “resynthesis” to take a sound and recreate it using additive synthesis (e.g. Hartmann Neuron, Alchemy), but I wouldn’t probably call this AI/ML.

Well it’s not midi but what you’re describing is similar to this approach: https://magenta.tensorflow.org/ddsp

arXiv link for more context: https://arxiv.org/abs/2001.04643 and blog posting with audio examples: https://magenta.tensorflow.org/ddsp

The skill of making music well enough that I can trust a machine to emulate my style is unfortunately absent in me. Your response is not only patronizing but also loaded with false premise. It's also apparently wrong: https://magenta.tensorflow.org/datasets Thanks for your contribution.

Here's an open-source project from Google Deepmind's Magenta Project, that does exactly what you want. https://magenta.tensorflow.org/onsets-frames

If you use Ableton Live you should check out Magenta Studio https://magenta.tensorflow.org/studio/ableton-live/

Cause you can do things like this https://magenta.tensorflow.org/blog/2018/12/20/style-transfer-js/

Some more information about this project: blog: https://magenta.tensorflow.org/pianogenie pdf: https://arxiv.org/abs/1810.05246 youtube video: https://youtu.be/YRb0XAnUpIk