https://github.com/yuval-reshef/StreamVC

Unofficial implementations of StreamVC

Are kidnappers and con-men a huge under-served market that Google is hoping to serve ? Deep Fake videos not convincing enough to serve the need of fraudsters ?

I am totally against regulating AI but shit like this gives fodder to the other side.

In this work, we propose a light-weight (~20M param.) causal voice conversion solution that can run in real-time with low latency on a commercially available mobile device. The key design elements are: (1) using a causal encoder to learn soft speech units; (2) injecting whitened f0 to improve pitch stability without leaking source speaker info.

In our later V2 version, we found that f0 rescaling followed by a NSF-style harmonic-plus-noise conditioning (as is done in RVC) results in better quality.

I know of one: transgender people often would like to alter the timbre of their voice and spend a lot of time training their voice. At least for online scenarios, this can just do it.

But other than that AI voice altering research seems like it benefits mostly scammers? I’m just wondering what they tell themselves they’re doing. I didn’t see this in the paper.

Also allows people uncomfortable with their natural voice, in particular transgender people, to communicate closer to how they wish to be perceived. Or even for someone to use their own natural voice from previous recordings if some temporary or chronic disease/disorder has impaired it.

There are probably a bunch of creative applications - like doing character voices for a D&D session or reading an audiobook. Obviously depends on the preferences of those involved, and many will currently dislike it on the basis of it being AI, but I think over time we'll see the tech integrated in interesting ways.

I imagine the majority of the use will be in entertainment/memes/satire - joining a call with an amusing voice on, or the equivalent of Snapchat's face filters. Not something critical that we couldn't do without, but still a fun application.

I don't see much benefit to kidnappers in this; if you just need to send an anonymous message without much concern about flow and latency, text or traditional TTS is fine.

Heck, I can even see broadcasting uses. Imagine if every on-air personality had good target files made ahead of time, so then when they catch a cold, production runs their lapel mic feed through this, using the "good" target sample, and remove all the congestion and raspiness.

> I am totally against regulating AI but shit like this gives fodder to the other side.

You think anonymity is so universally hated that it's actually bad PR for leaving AI completely unregulated? No other problems with AI that you can think of, and also no good reason why someone should be allowed to be anonymous?

It’s not a desire I ever had. But maybe people are different?

Alternatively, building the solution was so much fun that the question of whether this is a problem that should be solved was never asked.

The second was we got a very enthusiastic video spokesperson but unfortunately she has a very thick non-american accent and this can help us alleviate it.

But the prototypical legitimate use case (which we needn't be excited about), is a voice over artist leasing their timbre instead of their time so that new text can be made to sound like them without their being actively involved. If it were to become mature (which doesn't seem close, from this example), it would be a big step up from existing phone tree voice assemblage and would open the doors for dubbing, animation voiceover, harmonization, and ADR in commercial sound and film.

Gender masking or general anonymization aren't really served by this, as you don't need to adopt a specific target timbre to deliver on those. There are other techniques that work perfectly well for those uses, some that have already been around for ages.

I really believe that we are entering a "golden age" of fraud. It will be crazy.