DiffRhythm: Fast End-to-End Full-Length Song Generation with Latent Diffusion(aslp-lab.github.io) |
DiffRhythm: Fast End-to-End Full-Length Song Generation with Latent Diffusion(aslp-lab.github.io) |
That said, wow. An end to end FAST architecture that can infer a 4.5 minute song in 10 seconds is a compelling thing. I didn’t see if we got open weights, but my guess is that this is not crazy challenging to train, and some v2/v3 versions of this are likely to be good-to-very-good.
I suppose it might be because it's latent diffusion.
As an amateur musician, I'd like tools that help me be more productive musically - those that complement my skills (whatever they may be). All the things you mentioned above, namely, ability to score a melody via a simple hum, transfer to various instruments, generate proper responses to calls, generate melodies within a framework, etc., all these would be super valuable to me.
I'm an OK guitar + bass + keyboard player, I'd LOVE to have an AI assistant that accompanies along. That would make my own jammin' so much richer.
I dont think we have seen the end of AI-driven tools in music-tech yet. I'm cautiously hopeful.
That said, there is a difference to images in that music also has a "symbolic" level to it that is closer to text than images [1]. There's other work out there that uses LLM-type tools for direct melody generation (no audio). And of course, there's lyrics. I do expect commercial tools to start integrating all these capabilities gradually, it's just a matter of time.
[1] I guess there's also vector images (like SVG) - I've seen work in generating those as well, though it's less mature than directly generating pixels.
Story Jam lets you design chord progressions without needing to know about music theory, instead offering intuitive terms like "lightness", "darkness", "drifting" and "roaming". They mean about what you think they mean.
I'm planning a "Show HN" post for tomorrow morning EST with more details. But you can get the sneak peek here :)
“Pop punk with prog rock time signatures“ is a funny idea, but it’s not interesting to listen to when there’s obviously zero intention behind it.
Lots of sour grapes comments from folks. Too bad. Not what I expect out of Hacker News. Glad people are pushing the technological envelope and exploring this space despite the strong negative emotions.
absolutely crazy
I wonder what the hyper-capitalist's end game looks like. One giant company that covers everything with one man sitting at a dashboard, tweaking parameters? Is that one man even necessary?
I wonder what our plans are for when "the economy" prefers to do it's thing without us. Writing poems all day? What capitalist instrument will provide "money" for us to spend in this giant machine?
So they just hate humanity in general then.
If you think I’m being harsh, I have my reasons as a professional musician to critique these things in an unflattering light because they are my competition. Thankfully actually “generated” AI music is trash. Copyright is problematic in the US, I admit, but tech bros using copyrighted material to train programs to put us out of business - without paying a penny which even Spotify doesn’t per stream - yeah, I’ll have some disdain about this scenario and I feel it’s justified.
Just because you can doesn’t mean you should.
Ty it don’t think peasants were listening t to Bach, do you? Only the extraordinarily wealthy could afford to have music as anything like an every day thing.
We've done these things ourselves for hundreds of thousands of years. As we are increasingly convinced to buy them for convenience we loose the very things that make us know our connectedness.
So ya, there are real problems caused by the convenience of technology
Granted, many people are benefiting from these tools (myself included) but at some point a lot of us are going to have to find a new job (assuming the progression continues unabated), and I'm not sure what new jobs are going to exist when LLM coders replace many or most of us.
If you don't enjoy composing music, just don't do it, and give it to someone who does, and has the experience/knowledge/culture/practice/gut to do it.
Well it has the benefit of being true.
Tell me one example how music gen in any way benefits anybody to the level that is worth putting out of business the last few artists that make ends meet?
We've created machines to replace humans doing things humans enjoy doing. Leaving the drudgery machines were supposed to eliminate to be done by humans.
There's at least an order of magnitude more people who enjoy making music than there are people with the actual skill/talent to make music. Music generation AI is an absolute blessing to the untalented among us who'd love to make a song in a certain style or with certain lyrics but lack the time, talent or ability to do it ourselves.
But don't mistake one thing for the other: how is it different than, say, being Emperor Joseph II asking Mozart in Vienna to write an opera for him?
Mozart wrote the music, not Joseph.
Similarly, you can hike across France, from South to Britain for several days. Or you can take the train. Or a car, alone, or with a driver. Or a plane, in the pilot or the passenger seat.
You'll get in the same place in the end. The experience will be totally, fundamentally different for you, as well as for others.
I'm a musician myself, but I sadly suspect that most music made today "benefits humanity" very little... Is music making always a net positive? If nothing else, these tools will allow more music will be made.
Yes because the act of making music, even not very good music, is what has value. Music generated without human input has no discernible value.
> If nothing else, these tools will allow more music will be made.
By machines that, as far as we can tell, take no enjoyment from making it. And eliminates any possibility of emotional connection between the artist and the listener. Which is the entire source of music's value.
So, feel free to criticise capitalism and how inhumane it is, but don't anthropomorphise it by ascribing human emotions to the system.
For example, I tend to think of "composition" and "synthesis" as two very different topics.
One VST could spit out chords or melodies (not a common VST) whereas another could render those sounds (very common VST)
Old joke about airplane automation:
In the future there will be just one pilot and a dog in the cockpit. The dog is there to bite the pilot if he touches anything.
So where is it going? Why: the end.
But this is also where Gandolf says, “end?”
We would be better off if the other 99.9% didn't have worry about making ends meet, than if we do whatever it takes to keep the status quo of the 0.1% intact. That does not only go for artists.
Check how many ai songs are uploaded daily to spotify and any other platform.
And show me a few of those artists that hate making music and welcome ai.
These API composers perform a task many humans want to do. And there are roughly zero consumers of music saying "you know what's missing from the music market? music made with no human input."
This strictly serves capital. The goal is to destroy more artists livelihood to marginally increase the wealth of already wealthy people.
By the same logic synthesizers shouldn't have been invented that allowed people to make advanced sounds without tediously learning an instrument first, consumers should remain priced out of microphones and editing software, etc.
Like I said, I am not trying to feign ignorance on the drawbacks of the tech which is very real and far from negligible. I am not a tech bro AI maximalist. I just do believe that hyperbole will not put the djinn back into the bottle, and pretending like there isn't a real market between nothing and paying or being a composer isn't adding anything to the conversation.
Doesn't seem like much you could actually tweak beyond the short style queue and lyrics. To really be customizable you'd need a method to tweak the generation to inform the tone, inflection, and flow of the music at any given point to satisfy what you wanted to pair it with.
Songs have lulls and swells to go with the tone and emotion you're trying to create/communicate these are just strings of lyrics over boring barely connecting backing tracks.
Let’s follow the AI and automation craze to its eventual conclusion - automations everywhere, humans are either employed in automation industry, or are unemployed at a massive scale.
Stable jobs are replaced by ever-optimized gig economy for some, and chronic poverty for others. For there to even be economy - the massive underemployed population subsists on government welfare.
Cynic in me thinks that all of the wealth generated by enormous productivity gains resulting from automation will not find its way towards population displaced by it. Those cashiers, toll booth, and warehouse workers did not find themselves in much more lucrative careers - I don’t see why it will be any different for truck and cab drivers who will be joining them in the near future.
If you see a future where these people who suddenly found all this extra leisure time o. Their hands and no income - are somehow blossoming in creative directions and realizing their own potential - I’d like to have it painted for me, as it all looks pretty bleak to me. Just not quiet sure of the timeline.
Best I can come up with is an emergence of some kind of counter-cultural protest market where people buy and sell “made by humans” products, and are continuously attacked by various regulations originating from mega corporations who captured the government.
Empirically, that's not true.
Unemployment was at an all-time low after most of those jobs were eliminated, and wages after adjusting for inflation continued to rise in real terms.
Equally bad is anecdotal evidence, but I’ll drop some anyway. For a while now I am observing a crisis thats, admittedly subjectively, easy to see - but is somehow absent in those empirical sources citing economic accomplishments. An indirect evidence of what I am talking about - is crushing defeat of democrats/establishment in last election, following among other reasons, quite a backlash for boasting about said accomplishments.
But rather than picking issue with one of my points - I still would like someone to describe the counterpoint to my dystopian expectations - where, for example, would all those professional drivers I mentioned earlier go?
Ps. Oh speaking of statistics - remember Greenspan’s “there’s no real estate bubble, there’s froth in individual markets” right before 2008 financial crisis? It be funny like that, sometimes much derided common sense is all you need /shrug.
> People that never considered the value of artistic process until it was the topic du jour unilaterally decided that it was inefficient, oppressive, complex, frivolous, and unfairly inaccessible to those that hadn't put any sustained effort into developing theirs.
This is eerily reminiscent of what's happening inside the USA government & administration today...
For sure! After all, what could be more democratic than a monthly subscription that could get snatched away at any moment - and clearly there's nothing more creative than pressing a button and waiting for 20 seconds!
> People that never considered the value of artistic process
One certainly learns of crazy things on HackerNews. Apparently people have never considered the value of artistic process, and not only that, but you also happen know that exactly.
> the topic du jour unilaterally decided
You're literally in this thread disagreeing.
> it was inefficient, oppressive, complex, frivolous, and unfairly inaccessible
Very interesting claims, too bad they were only stated in your imagination. That being said, your imagination I think is surprisingly close to my opinions! Let's discuss each point:
- it is very time-intensive to produce creative works of any kind, and indeed to perform any kind of mental work at all
- it does get pretty complex too, and because of this, some mental efforts are even shot down for being too frivolous (such as that bit of automation that is not worth making because it would never pay itself off)
- oppressive is a bit of an odd one, but if I think hard enough, I guess I can see how having to use the output of e.g. my work (software) can be oppressive
- same for unfairly inaccessible - lately there's been a trend where various services would only be available online, and the only contact you'd get is a self-service form or two. Maaaybe you'd get an AI chatbot to chat with. Certainly, to those with minimal to no tech literacy, this will be inaccessible and it will feel unfair.
> was merely the natural progression of artistic praxis
If only there was a way to disagree with this without being a dickhead!
> these tools have democratized creativity
How does one democratize an innate property of people? Surely you mean that they have democratized the production of creative works rather, and even of those only the less high-art ones, which I'm sure you never fail to point out when shown one?
> they're now broadly available to anyone willing to pay money for a subscription service that will obviously soon be a hell of a lot more expensive, or shell out a few thousands dollars for a top-tier video card that you almost certainly already have in your gaming rig, anyway.
And what happens after that? Artists will be like "oh gee, well I'm not doing this again!"?
> This is silicon valley progress
And also Hangzhou and Shenzen, China.
> and if you don't like it, you're a communist
Are you? You seem to be more of a raging idiot than anything to me at least.
This supposes that the music is the end goal, and the very point of my comment is that it doesn't always have to be, and in those cases "just don't do it" also means not doing whatever comes after.
Just as you state below, this doesn't replace creating music for the creation's sake. I don't believe it will, or should. It merely replaces having nothing at all, or having the 100,000th video with the same upbeat stock sound.
It's just that.. you can't master something you don't practice and understand. It's true in every single thing in life you do, sports, literature, maths, music, cuisine, kindness, etc.
If you don't like to compose music, why suffer this and even submit to the randomness of some computer program, rather than giving the opportunity to another fellow human to open your ears and your mind to what they appreciate doing?
You can generate your music if you like. It just cannot compare to something a human really did on her own, and invested of her desire, time, practice, research, even a beginner.
It's not a matter of being professional or not. The best musicians I know are not professionals, they all have a day job.
For every famous star for one given instrument, you have 10s of undiscovered/local better musicians that just are carpenters, cooks, painters, drivers, factory workers.
I think "stop not enjoying it" is a better line to take. Like with AI illustrations (where I'd much rather see a blog author's crappy biro drawings instead), terrible amateur efforts with some online 808 emulator or whatever would be more entertaining and interesting than AI output.
Wherever all the cashiers, toll booth operators, and farmers went after automation took their jobs.
New jobs are created, the people displaced have to migrate to them.
Is it fun for them? No.
Is it how the world works? Yes.
Technology thus far has a VERY VERY long and established role of creating more jobs than it eliminates.
See >95% of the population being employed in agriculture for tens of thousands of years and being reduced to about 5% over the course of 100 years (and civilization being FAR FAR better off for it).
Will that trend one day end? Probably.
Will it be doomsday for the plebs? Who knows.
Is it happening within a timeframe worth worrying about? Unlikely.
Creation is a human activity, charged with emotions, efforts, which are their own rewards, as much as the end-product, which is invested of this human (sometimes collective and not instant) effort and intention and creative loopbacks. Let's call that some kind of history (because the process did happen).
Generation short-circuits that entirely, as it happens at non-human speeds, and non-human scales. It's something _else_ entirely. You do get an end-product. It may be fun and useful for some; it sometimes is. However, you don't get the process, the collaboration and the inner transformation it comes with.
Adding: with two different end-products, the issue is then how they are perceived, received, appreciated and valued by those not "in the know" of how they were made. And that is both an artistic, aesthetic and economic problem. Generating soulless shit that isn't invested with a human sentiment miseducates people and destroys taste.
Models that are oriented around one-shot, text-only direction are pretty limiting in creative flow. This will hopefully continue to improve.
In the same sense that a music producer telling an artist to "make me a new song" or a new album is creation of music on the part of the producer.
It's not what most people think of when they hear "creating music".
This does not augment the music making process in any way, it simply replaces it with what might as well be a gacha game. There's no low-level experimentation, no knowledge acquisition, no growth, and you can't even truly say you made whatever comes out.
It's not a tool for music creators, it's a tool for people who want slop that's "good enough".
"Giving it someone who does" is also an opportunity to socialise and grow a mutual understanding of said music desire. That's sometimes even how collaborations start. But that's not on short notice...
Perhaps generators could be also seen as some kind of introductory instruments to wet the appetite of becoming musicians?
Bullshit. These "tools" replace the creator, who now has no meaningful input to what gets created.
The composer is the AI. Not the person who spent 10 seconds typing a prompt.
Making an activity in which the primary limiting factors for most people are the time, knowledge, and effort required (as opposed to expensive tools) into an effortless slot machine pull is enfeebling to human creativity and agency. Who will spend the hours of making bad music to get to the point where they become good if they can just rely on something else to generate music that's "good enough"?
There's something to be said about all this which is related to AI generated images that I rarely see brought up: people with specific skills play roles within groups, so AI making their hobby that they dedicated so much time to more easily accessible makes them lose social value, which might make them quit altogether.
The common response that "people should make art because they love it, not for attention" is a prescriptive statement that supposes there are more or less "pure" forms of performing an activity and also ignores that art is a form of communication.
I agree that the slot machine pull of current models is tedious and boring. I look forward to models/systems which better facilitate more creative control, directed exploration and iterative refinement.
Accurate value judgements in this case.