Show HN: Midjourney for Language Learning Videos

5 points by cmuguythrow 1 year ago | 4 comments

Demo: https://youtu.be/fXj5VYYaoeY Website: https://plusonechinese.com

Hey HN, we're Patrick and Thomas and are building an app that uses generative video to teach languages (currently Mandarin only).

Background: We think the "Comprehensible Input" theory of language learning has a lot of merit - TL;DR: consume native content that you can understand a high % of and pick everything else up as you go with context. For more detail see https://refold.la/roadmap/ or https://en.wikipedia.org/wiki/Input_hypothesis However, it is quite difficult to find content that is interesting to you, at your level, and in a format you can study it. Generative AI can help bridge this content gap.

What we've built: An AI video agent that takes your vocab level and a short prompt and spits out a ~2-minute video. The video will have dialogue that is 85% comprehensible at the given vocab level, making learning the remaining 15% of new words easier and more effective.

For example, this video was generated with a 2,150 word vocab and the prompt “Bringing a pet capybara to the public pool”. https://youtu.be/B7zRNl60hPY

Video Details: From that simple prompt we make a story, break down the characters and environment, write the script and ensure it’s properly levelled, break it down into shots, bring those to life with tts/image/video/lipsync, and stitch it together with music. As far as we can tell, this is the first end-to-end automated story video workflow of its kind, with no manual steps or cherry-picking, no choosing from character templates, etc.

The app is free for watching videos and studying flashcards, and doesn’t require an email to sign up. After the first free video generation, making your own videos is $15/mo for 1 video/day. Excited to hear everyone’s feedback!

feelingsonice 1 year ago |

First end-to-end automated story video workflow of its kind Videos like these [1][2] are a dime a dozen and have been out for years. Surprised you think this is completely novel.

[1] https://www.youtube.com/watch?v=tzE7TYwAYq4 [2] https://www.youtube.com/watch?v=az7KfOQkMu0

tigger0jk 1 year ago | |

Patrick here. Yeah there's a lot of AI video out there. There are also plenty that are more story-driven better ones nowadays. https://runwayml.com/gen48

None of these are end-to-end automated though. Even for a video without a story like the Harry Potter Balenciaga style ones, there's a lot of manual cherry-picking and manual editing going on. Here's a process example for that type of content that looks quite automateable https://www.youtube.com/watch?v=TGD8zKvRxc4 - but both 1. No one _has_ automated this and 2. It's much more difficult than it looks because of the manual cherry-picking part, and the story, character/enivornment consistency, etc.

I am really looking for another instance of "type text, get story video". I do think it's a bold claim that we're first but I haven't seen a counterexample yet.

hitchhiker2526 1 year ago |

Curious, why Mandarin only? Is there something about the language that makes it more feasible or more attractive to the business?. FWIW I would consider using something like this if it could teach Japanese.

tigger0jk 1 year ago | |

We plan to expand to all languages (the only real limits are languages with enough training data for LLMs / TTS models, so all popular languages)

Mandarin is just first for us because:

1. Thomas was already learning it

2. We can talk to users in English

3. We have some native-speaking friends who helped early

4. Has excellent support in AI products (second only to English)