GPT‑Live‑1 in the API(openai.com) |
GPT‑Live‑1 in the API(openai.com) |
I think education is a very underexplored area for these live conversation models. Yes you can just use ChatGPT Live but that's freeform and unstructured, doesn't have a curriculum or can present supporting visuals, etc. On a grand scale if you can give children their own personal individual tutor rather than relying on group teaching alone, there could be a huge jump in successful education outcomes.
I did think of implementing this conversationally, but tbh it has always been too expensive thus far, I gotta retry with GPT-live-1, I tried it with elevenlabs before but it wasn't live enough and the models were not intelligent enough.
[1] https://river.berlin/projects/german-learning-helper/ [2] https://river.berlin/projects/flashcard-generator/ [3] https://river.berlin/projects/german-cases-trainer/
When you use ChatGPT Live it's instant which is great, although the realtime transcription still kinda sucks, especially if you're a newcomer to the language so you're making mistakes. I'll constantly get responses to something it thinks I said but I didn't say, which is a real hard blocker for a language learning app.
I think what I'll land on is Scribe v2 (the full thing, not realtime) transcribing turns - it is exceptionally accurate for this - and then just feeding that text direct to GPT Live.
Thanks!
But, it's close! You can control their pronunciation, make them speak slower/faster, and obviously great at anything text, so many use cases work great for language learning with LLMs. Just wish they solved this last mile thing too!
Built my own app, currently focused on having it generate stories for me at my level, and also doing FSRS flashcards, using words that i lookup.
What are you using for the voice AI?
Right now I'm using Gemini as it seemed the best value, but I think it could be improved.
Also, how's the learning for you been so far? I've only been at it a few weeks but this method seems to be superior than other things I've tried
However, for voicing sentences I use murf.ai, which seemed very nice for korean.
I tried to talk to the openai version in (my bad) korean, and it responded in japanese :D
Group classes are great for language schools (more $) but can be bad for students:
hard to match levels speaking to non native speakers reinforces bad habits
Congrats on the launch here. I've been messing with this over the last few hours- super super cool. I was excitedly awaiting this hitting the API, because ofc there wasn't a super high fidelity option for drop-in voice interface in front of a given harness. This is blowing me away so far!
(Side q, is there a single place one can watch for updates on the API- that actually covers everything that changes? IIRC there have been a couple of additions that you've tweeted- but never hit the API changelog ;] )
Not to mention all the current "telephone bots" applications that could benefit from something that has actual reliable STT and can accurately grasp a number you tell it first try, or hear a natural language description of what you want and immediately bypass listing the entire menu of options one by one.
You obviously need humans but they can be freed up to deal with the more complicated cases.
Try to contact Anthropic or Google support, they clearly don’t think so.
Another question is: can the model interrupt you and ask questions right away? What if you're incorrectly defining something and then building up on it? Would the model interrupt you right after the incorrect definition or after you've already finished your explanation?
Anyone else hack something together with HA / the Voice PE yet?
Looks like it needs a second model to do function calling, which gpt-realtime-2.5 didn't, and the Voice PE XMOS chip's audio pipeline might not be a great fit for full duplex back and forth, like what gpt-live-1 now supports.
I don't know if they are having some troubles with their demo environment due to the volume of requests in this specific moment, but I'm very hesitant to put something like this in production if it fails with this trivial example.
Honestly though they need to hire whoever did those Google ads back in the day.
Is somebody from OpenAI hiring? I can show how I use it to learn German, among other very interesting usages.
Also show proper excitement etc... I think also a lot of real users could do better.
It feels like they aren't real users of their own products...
However, this open source project https://github.com/chatbotkit/platform/ does and you can plug OpenRouter or OpenAI keys straight in while keeping your integration work generic. The only downside is hosting it yourself but it is just docker compose up.
I played around a bit with this in Codex when it became available but even when you have Fast mode + Light reasoning, the mere idea that it passes off actual work to background sessions even for "change this line here" makes it a really frustrating experience.
You can say "Update config here" then wait 2 minutes then finally it comes back, and most of those two minutes was overhead of agent<>sub-agent communication and passing the work, instead of just, you know, do the thing.
I'm eagerly awaiting for this to get ready though, because being able to use tools like Houdini, Unreal Engine and Blender over MCP with this fast voice mode makes for great video game development environment, where you can playtest the game and talk with Codex at the same time, asking it to update stuff on the fly, granted you've setup things correctly.
For a single line change that could be completed in maybe 10 seconds if the agent didn't do the whole communication overhead dance, half a minute is a long time to sit and wait during playtesting to just update some parameter.
> And you can talk about the next change while the previous one is being implemented.
Yeaah, that's not how I work with agents in general, we work on one thing, do it properly and then clean up, then refactor, then testing, more refactor and so on, until a thing is 100% nailed down and properly implemented and then move on. I don't know how people can work on multiple things at the same time, unless they're really simple tasks or small projects. But almost nothing in medium/long-term game development is that simple.
It baffles me that the labs can't come up with more exciting use cases for voice api.
Nonetheless, in complete honestly I do also have a German tutor who I see once a week for 50 minutes, I am very reliable on completing my work though, the "progressbars" in my flashcards app do keep me motivated.
Learning a language is really hard and takes years, but mentally I am convinced, that if the progressbars in the flashcard app I use reach 100% and also in my German cases app, that I will get closer to speaking perfect German, this keeps me motivated.
Like for me it's the part in The Wind in the Willows where they're on the cart, before Toad becomes a car fanboy. I asked ChatGPT to stretch that part and make them stay on the road for longer, and it kinda did an OK job tbh
I have special instructions in my AGENTS.md to bypass running the full QA suite for small contained changes, and to run a targeted one instead. And if an error passes through, it will be caught the next time the full suite is run.
I have both a Rust and a Python project, the Python one has a full QA suite, the Rust one is much more bare bones, and surprisingly, I can have Astra implement a small Rust change in 10 seconds, but not a Python one.
No, it's a runtime parameter, imagine "walk speed", with zero tests as it's a 100% authored experience, tests live elsewhere.
The one line change takes 2 minutes because of the communication overhead, which I clearly stated in my previous comment. One turn having one reasoning block, one tool call and one final reply, is obviously gonna be faster than one turn with a sub-agent which is at least two reasoning blocks, two tool calls, two final replies, two messages passed between the two agents. I'm not sure why this be surprising to anyone that it'd be slower.
I do that when I know exactly what I want, and use ChatGPT voice when I need to explore the solution space.
Yeah, but then it's no longer GPT-Live-1 which is what we're discussing here, the point is the bidirectional voice mode and the faster response times...