Llasa: Llama-Based Speech Synthesis(llasatts.github.io) |
Llasa: Llama-Based Speech Synthesis(llasatts.github.io) |
Llasa-3b (https://huggingface.co/HKUSTAudio/Llasa-3B) came out before Orpheus (https://huggingface.co/canopylabs/orpheus-3b-0.1-ft).
> it's the exact same thing.
They're very similar, but they're not the exact same thing.
Llasa uses xcodec2, a much simpler, lossless 16khz wav codec. This makes it superior for one-shot voice cloning.
Orpheus' 24khz snac codec is lossy which makes it difficult to use for zero-shot cloning as the reference audio gets degraded during tokenization. You can test this here: https://huggingface.co/spaces/Gapeleon/snac_test
But when finetuned on 50+ audio samples, it produces much cleaner 24khz audio than Llasa, and the snac model is much easier to run on consumer hardware than xcodec2 (87t/s for realtime speech, which can be achieved on an RTX3080 for example)
I really wish when new models were released that they would draw a diagram of all the layers and the tensor input and output sizes at each layer, with zoom in/out capabilities if needed using D3.js or whatever visualization framework if needed. Every single layer should be on there with its input and output sizes.
These one-sentence descriptions, and approximate block diagrams with arrows pointing at each other are never enough to understand how something is actually implemented.
You can also build a custom version of llama.cpp that writes out the ggml compute graph. What's irritating is that hugging face didn't add it to their GGUF file viewer.
I just wish that new research would always spell it out in full instead of these silly block diagrams labelled with just e.g. "Cross Attention" and not the exact parameters, number of heads, layer sizes, etc.
Also some of these diagrams use a + for concatenation and some use it for addition, that's another headache to figure out, having layer sizes would make it clear.
1B is actually huge for a TTS model. Here's an 82m model with probably the most stable/coherent output of all the open weights tts models I've tested: https://huggingface.co/spaces/hexgrad/Kokoro-TTS
But if you mean zero-shot cloning, yeah they all seem to have those slurred speech artefacts from time to time.
Shouldn't there be some hardware module be available similar to how Alexa, Siri and Google do it?
Whith a ring buffer detection the word without recording everything?
Zonos uses 128-float embeddings for voices and it seems so much nicer. Because you can just mix and match voices without changing the model.
What are people using to upsampling back to 44,1 or 48 khz? Anything fancy?