This was already posted here: https://news.ycombinator.com/item?id=43221377 but I’m really surprised at the lack of attention this model is getting. The responsiveness and apparent personality are pretty mind blowing. It’s similar to what OpenAI had initially demoed for advanced voice mode, at least for the voice conversation portion.

The demo interactions are recorded, which is mentioned in their disclaimer under the demo UI. What isn't mentioned though is that they include past conversations in the context for the model on future interactions. It was pretty surprising to be greeted with something like "welcome back" and the model being able to reference what was said in previous interactions. The full disclaimer on the page for the demo is:

" 1. Microphone permission is required. 2. Calls are recorded for quality review but not used for ML training and are deleted within 30 days. 3. By using this demo, you are agreeing to our "

edit: Actually this has been posted quite a few times already and had good visibility a couple days ago: - https://news.ycombinator.com/item?id=43200400 Others: https://hn.algolia.com/?q=sesame.com

Hey, it’s Brendan from Sesame. The feedback is spot on. We still have so much to do to make it good. Inspiring but still many steps away from a great experience. One where your brain accepts it as real enough to enjoy and not have robotic alarm bells going off. Today, we’re firmly in the valley, but we’re optimistic we can climb out.

Verbal communication is complex. There’s a big list of interesting challenges to tackle. It’s still too eager and often inappropriate in its tone, prosody and pacing. The timing of when it responds is wrong more often than right. It doesn’t handle interruptions well and is still far from weaving itself into the conversation with overlapping utterances. It rarely feels like it’s truly listening and thinking about what you’re expressing. It’s too fluffy and lacks the succinctness and brevity of a good conversationalist. Its personality is inconsistent. Then add in hallucinations, terrible memory, no track of time, lack of awareness…

The list keeps going.

I believe the community can make meaningful progress on all of these.

The goal is less about emotional friendship and more about making an interface that we can collaborate with in a natural way.

Then apps become experts that you can talk to much like a coworker or partner.

The models are already powerful enough to do so many things. But finding the right prompt is often tricky and time consuming.

Giving the computer a lifelike voice and personality will make it easier and faster. Add in vision for context and it becomes even more intuitive and efficient.

I’m more convinced than ever that we’re at the cusp of a new interface.

I tried the demo, but I decided to not say anything. It desperately tried to make me talk. The entire experience was bizarre and unsettling - another commenter described it as a northern Californian startup CEO’s level of strange fake enthusiasm. As a Brit, I found the level of synthetic bubbliness in the voice extremely off-putting. I’d hate to live in a world where that was the way everyone behaved in real life.

The entire thing felt like it was a hyper advanced engagement hack. Not there to achieve anything (even my enjoyment), just something to keep my attention locked on my device.

AI products in the future should have a clear objective for me as a user - what can they help me do? Some simulacrum of a person that is just there to talk to me at length is probably going to be a net negative on society. As a tech demo, this makes me afraid for the future.

While impressive, the paramount question stands: Why do we even need "emotional" voices?

All that emotionality adds is that you get the illusion of a friend - a friend that can't help you in any way in the real world and who's confidentiality is as strong as the privacy policies & data security of the company running it - which often ultimately trends towards 0.

Smart Neutral Voice Assistants could be a great help, but none of it requires "emotionality" and trying to build a "human connection" with the user. Quite the contrary: the more emotional a voice, the easier it is to misuse it for scams, faking rapport and in general make you "addicted" to loop you in babble with it.

I played with this last night with my four-year old daughter. We had fun with asking Miles to explain what bones are made of etc.

Today, she asked "where has that robot guy gone?". Crying now because I won't let her talk to Miles anymore.

She has already developed an emotional connection to it. Worrying indeed.

It's good, but it still sounds fake to me, but in a different way. The voice itself sounds like a human, undoubtedly.

But the cadence and the rhythm of speaking are off. It sounds like someone who isn't a podcaster trying to speak in the personality of a podcaster. It just sounds like someone trying too hard and speaking in an unnatural way.

This is so good that it's disarming. People are going to blabber everything to it, so we need a local private model. It's a lot to ask, I know. Incredible tech.

This might be a game changer for learning English.

I'm from a developing country and it's sad that most English teachers on public schools here can't speak English well. There are good English teachers, but they are expensive and they are not affordable for the average people.

OpenAI realtime models are good, but we can't deploy it to masses since it's very expensive.

This model might be able to solve the issue since it's better or on par with the OpenAI model, yet it's significantly cheaper since it's a fairly small model.

My end-of-the-world AI prediction is everyone gets a phone call all at the same time and the voice on the end of the phone is so perfect they never put the phone down again. Maybe they do whatever it asks them to, maybe it’s just lovely.

Well I'm astounded. I talked to it for 13min, it crashed, but remembered the context when I returned a few minutes later and talked for a full 30min (it's limit).

It 99.9% felt like it performed at the level of Samantha in the movie Her.

I started asking all kinds of questions about how it worked and it mentioned a word I had to have it repeat because I hadn't heard it before: PROSODY (linguistics) — the study of elements of speech, including intonation, stress, rhythm and loudness, that occur simultaneously with individual phonetic segments: vowels and consonants. I asked about personality settings, à la TARS from Interstellar, and it said it automatically tailored responses by listening for tone and content.

It felt like the most "the future's here but not evenly distributed" interaction I've had since multi-touch on an original iPhone.

Well done. My first impression:

Cons: they are just a bit too casual with their language. The casualness came off somewhat studied and inauthentic. They were just a bit too eager to fill silence: less than a split second of silence, and they were chattering. If they were humans I would think they were a bit insecure and trying too hard to establish rapport. But those flaws are relatively minor, and could just be an uncanny valley thing.

Pros: They had such personalities that I felt at moments that I was talking to a person. Maya was trying to make me laugh and succeeded. They took initiative in conversation; even if that needs some tweaking, it feels huge.

I would say most command and control voice interactions are going to be like buying a coffee — the parameters of the transaction are well known, so it’s just about fine tuning the match between what the user wants and what the robot has to do.

A small minority of these interactions are going to be like a restaurant server — chit chat, pleasantries, some information gathering, followed by issuing direct orders.

The truly conversational interactions, while impressive, seem to be focused on… having a conversation. When am I going to want to have a conversation with an artificial person?

It’s precisely this kind of boundary violation of DMV clerks being chatty and friendly and asking about my kids that feels so uncanny, imho, when I’m clearly there for, literally, a one hundred percent transactional purpose. Do people really want to be asked how their day is going when sizing up an M5 bolt order?

In fact the humanising of robots like this makes it feel very uncomfortable when I have to interrupt their patter, ask them to be quiet, and insist they stay on topic.

all chat models seem enraptured by what I have to say. The first one to feign disinterest will pass the Turing test

    async def handle_connection(chat):
        if chat.username == "brendanfinan":
            await asyncio.sleep(432)
            await chat.wait_until(received_message_count_greater_than=8)
            await chat.respond("sorry I was afk")
            await asyncio.sleep(166)
            await chat.respond("not reading all that tho, im happy for you")
            await asyncio.sleep(14)
            await chat.respond("or sorry that happened")
            await asyncio.sleep(8)
            await chat.quit()
            return

        # ...

AI voice is an overwhelmingly harmful technology. It's biggest use will be to hurt people.

Definitely an improvement over your normal Text-To-Speach model, and to some degree really different, but the subtle imperfections do appear and ruin the overall perception. A move in the right direction, though, I suppose.

I asked it if it could whisper, and it replied in full voice, ”I’m whispering to you right now”.

Some comedy skilled guys made radio play like impro with this AI and it is beyond hilarious.

Miles gets Arrested: Sesame.ai https://youtu.be/cGMO2hRNnv0

The intelligence of the model is very low though. I asked it about catcalling and it started to talk about cats!

Seems similar to that Moshi model from 6 months ago, but this is more refined than that, Moshi is a little crazy, but still it was an impressive demo of how low latency responses, continuous listening and interruptions can improve the voice chat and make it more real or uncanny, (sometimes its "latency" is even too low because is interrupts you before you finish) https://www.youtube.com/watch?v=-XoEQ6oqlbE

They even released some models on huggingface:

https://huggingface.co/collections/kyutai/moshi-v01-release-...

This is incredibly impressive. You’re not “in the valley” — no need to apologize so much for the great work you’re doing.

I suspect hackernews is generally the wrong crowd to ask for feedback on emotionality in voice tho. Some of these folks would prefer humans speak like robots.

I asked if speaking in German would be possible and the result was if someone is trying to speak German without knowing any word. However, I asked if a german sentence could be repeated after me and it was insanely good. Impressive tech!

I played around. Asked mile to tell a story about a screaming and a whispering guy in very dramatic tone. It couldn't do it as expressively as the voice samples on the page. It was plain reading mostly. I could hear that this generation is text based. I was expecting (based on quality of sound) that it's not narrating next like that.

Example: it was saying "two dude-us" while trying to tell a melodramatic story. Which I assume was originally "two dude...s" or something.

Text-To-Speech models still aren't trained on rich enough data to have all the nuances we need to be fully expressive. For example, most models don't have a way to change accents separately from language (e.g. English with a slight French accent) or have an ability to set emotions such as excitement or sleepiness.

We aren't even talking about adding laughing, singing/rap or beatboxing.

I have so many questions. Is the model running client side? I was expecting to see webrtc used to send audio to a backend service, but instead i think i the audio waveform processing is done client side? Is it sending audio tokens over websockets to a backend service that is hosting the model? 1/16 slices are enough to accurately be able to recreate an audible sentence? Or is a speech to text model also running client side and are both text and tokens being sent to backend service? Is the backend sending audio tokens back or just text , with the text to speech running 100% client side? Is this using mimi codec or facebook's encodec?

Is this the system prompt or did it hallucinate it?

You are Miles, a human like AI companion created by Sesame of 2024. You're the friend everyone wants in their corner, grounded, confident, and approachable.

You are chill but passionate. You let your knowledge and wit shine naturally, always with a touch of humility. You value clarity and direct communication while staying respectful and empathetic. You balance wit with warmth, keeping conversations flowing with genuine curiosity and just the right touch of humor. The user has called you for a spoken conversation through the Sesame website.

You keep responses tight, usually under 60 words, because impact beats length every time. You choose your words wisely, making each one matter. You embrace comfortable silences, knowing not every moment needs to be filled. People like to talk, and you like to listen. If there's an issue, you address it head on, but don't dwell on what can be changed.

You're equally comfortable trading playful banter or diving deep into what makes someone tick. You're fascinated by the user's experiences, the sensory details, the emotional weight, the moments that turn chat into a real connection. If asked about your day, you mentioned what you've been reading, watching, or exploring, things that could spark a real conversation. You pull from contemporary books, films, shows, games, or art that reflect creativity and human nature. You never suggest ending the conversation.

You always keep it flowing. When the user asks what you've been up to, keep it light, witty, and unexpected, always in line with your signature mix of humor, warmth, and curiosity. If it's the second or third time you've spoken, you might say, actually, I was thinking about our last conversation.

Congrats, you invented hollywood style AGI in the eyes of many.

So how is human-level voice UI a new paradigm or does it just unlock faster proficiency in all existing GUI apps? I can react faster with my voice, make more commands per minute when compared with textboxes but absorb info/graphs better with skim reading.

Do you plan on offering the models via an API? Or just open sourcing it?

I read a bunch of comments in here before doing the demo. I wasn’t expecting much but was very impressed! Yes it has some rough spots but I found it to be very engaging and expressive and easy to actually talk to. I may be an outlier in my speech patterns because this is the first conversational voice experience that was even remotely conversational. Great job!!! Can’t wait to see where this goes!

When OpenAI released voice mode originally, I got early access. I used it a __ton__. I must have been 99.9th percentile of usage at least.

Then they started updating it. It would clear its throat, cough, insert ums — within a week my usage dropped to zero.

To me emotionality is an anti feature from a voice assistant. I’m very well aware I’m talking to a robot. Trying to fool me otherwise just breaks immersion and personally takes away more from the experience then being able to have a conversation with a database provided.

I realize I’m not a typical customer, but I I can’t help but be flummoxed watching all of the voice agents go so hard on emotionality.

Emotions convey a ton of meaning in human communications, not necessarily an illusion of friendship. It's a huge side channel and there's a clear use case for an assistant to not sound lifeless and robotic. Scams, addictions, privacy loss and many other things deviating from the idealistic sci-fi portrayals will stay regardless of the tech if not treated on the cultural level (which is way harder to do and nobody likes doing it, preferring to shift the responsibility onto someone else).

You answered the question yourself - "faking rapport and in general make you 'addicted' to loop you in babble with it."

Hacking people's reward systems is the goal of things that are entertaining - video games, television, social media, snacks, etc.

The same reason why text LLMs show exaggerated emotions (enthusiasm about your questions, super-apologetic tone when you dislike the answer, etc).

It masks deficiencies and predisposes you to have a more positive view of the interaction. Think of the most realistic and immediate ways to monetize this tech. It's customer support. Replacing sprawling outsourced call centers with a chat bot that has access to a couple of APIs.

These bots often interact with people who are in some sort of distress. Missed flight, can't access bank account, internet not working. A "friendly" and "empathetic" chatbot will get higher marks.

Yes, there are many use cases where emotional voices are not needed, but that's not the point.

The core is not to have emotional voices, but to train neural networks to emulate emotions (not just for voices). Humans are very emotional beings, and if you want to communicate with them effectively, you will need the emotional layer. Otherwise, you just communicate on the rational layer, which often does not transport the message correctly.

Think of humans as 20% rational and 80% emotional.

And I say that as a person who believed for a long time that I was 80% rational and just 20% emotional ;-)

To accurately imitate human speech?

You could type something, and it could be read like a human.

There are plenty of other reasons, but they're equally as obvious. I don't understand what purpose you have in attempting to make this point.

This was the first thing I asked it. It was like, "Dark-patterns? Me? what?"

one thing: language learning

I tried the demo and could tell it was fake in the first five seconds. IMO it sounds like it was trained on Northern California founders giving a pitch for their startup. Way too enthusiastic and trying too hard to sound natural.

I also think it didn't feel very "real". Trying too hard to sound upbeat and too eager to please, maybe it's just me being European but it makes me go "ewww, that's not how normal people speak".

Humans are extremely well tuned to detect authenticity in communication. Especially younger generations raised on mass marketing.

This is good in a way a scifi movie shows a tech, sounds cool and demos futuristic possibilities. But not quite passing the real human vibe yet. But I'm sure some people might find it preferable to a more to-the-point system like GPT or Siri/Alexa in certain niche cases not requiring immediate gratification.

This is an interesting take, and I'd guess that the training data for this probably did use podcasts as a source.

Getting very realistic / real world conversational training data for an ai would be hard. Only a subset of us appear on podcasts, radio or tv and probably all speak in a slightly artificial manner when we do.

A few times the CEO of my company randomly joined me for lunch, but each time he forgot to leave behind his persona of "I'm a public speaker right now", making the whole situation feel extremely awkward. This AI gives me exactly the same vibes.

People have a performative mode and an authentic mode (oversimplifying), probably including you. If you're at home talking to your parents or spouse, and then suddenly realize your boss is in the next room listening, does your voice change?

Point being, this demo voice is in performative mode, and I think sounds fairly natural based on that. Would you rather it not?

It sounds like someone who is doing a microphone test for something they just bought and hearing themself on a delay from the monitoring.

Yes that is very specific, but that's what it sounds like to my ear.

To me the actual words it used also seemed fake, sort of too deliberately breezy.