Gemini Robotics 2 brings whole body intelligence to robots(deepmind.google) |
Gemini Robotics 2 brings whole body intelligence to robots(deepmind.google) |
It would be cool to have a robot that can be taught to drive the same way we might teach a teenager to drive.
But in a physical robot? Yeah, that thing is going to punch me in the face, eventually. Or worse.
Gemini doesn't remember almost anything after 2-3 follow ups. I have to paste the same "system prompt" at the top of each message and it still doesn't understand it.
For both manipulation and autonomous driving, google has invested in approaches with custom hardware, and off-the-shelf hardware, and a blend (which is what waymo is).
And Level 4 is already here, scaling up, while we're seeing the first real signs of Level 5 (Tesla FSD Supervised).
The progress here is staggering - I'm not sure why you're so cynical!
https://www.engadget.com/2225849/google-shuts-down-alphafold...
For businesses the bar for adoption is very low: If the thing can work repetitive jobs for 24 hours a day and replace 3 shifts, the purchase bar is nominally anything less than 3 x human salary if your budgeting horizon is 1 year. That's a high number, and probably fairly easy to achieve.
For homes, it's a very different bar. You'd have a hard time convincing most American families to purchase anything with a >$1000 price tag. Currently that's pretty much impossible for a humanoid.
I don't think we'll get household robots anytime soon. Hell, the only ones that can afford them will be the same that would hire human household help.
There is going to be so much pain from VLA malicious compliance. If the current gen of LLMs are anything to go by I can already see it being an hilariously massive problem. People are careless.
it depends. in order for my slow ass Roomba to clean my floors, I have to move a bunch of things out of the way and not use the room.
on the flip side, i think this "the robot cannot automate the whole task" thing is reductive too. suffice it to say, EVERYTHING matters, there honestly nothing that "honestly doesn't matter."
The difference between gpt 2 and 3 was insane. 2 could generate limericks when it wasn't repeating a word 300x. 3 could actually do some things. By comparison gemini robotics has hardly changed at all.
I will also point out that slow, non-fluid robotics is on a totally different level of difficulty from fast fluid motion. Asimov could walk pretty smoothly, but it didn't fall over because it used a very careful sequence that was never unbalanced; you could pause at any point without falling over. Move faster, like boston dynamics, and you need to account for the change in balance from your arms swinging... or rather, you need to be able to account for the rotational inertia etc from moving multiple masses along complex paths with multiple points of articulation at hundreds or thousands of times per second.
An algorithm to fold tshirts 90% of the time is easy. The cloth hangs down by gravity and you can just look for right angles (corners), find their coordinates with binocular matching, and move them to meet each other. Getting 99%, or folding them quickly, so that the fabric is actually moving instead of just hanging still- incredibly, incredibly more complex.
TBF fabric is much more difficult to simulate than a bipedal body is.
As I recall openai had mujoco playing soccer nearly 10 years ago. Obviously real world bodies are much more difficult but I'd be curious to learn why that is.
LLMs aren't themselves hurting anyone, no matter how much certain people like to pretend otherwise, whereas an AI in a robot with significant motors in absolutely can and will. There are reasons industrial ones live in safety cages after all.
Expecting them to be like inverse kinematic driven digital dolls is wrong because the optimization won't be for matching that but something like "net reduce energy consumption" which for electric motors in multi joint arms will look a bit odd.
Is this the new "x technology is a year away" ?
Just want to say, Deepmind is a great place to work and the only (Edit: one the few unique labs!) lab where you can move from large frontier models (Gemini), open models (Gemma), robotics (what you see here), science (weather, biology, more) and basically any other topic related to intelligence. It's really an incredible place to be, with incredible people. Consider joining! And thank you for the enthusiasm here.
We’d love to invite you to speak about your work next year at the AGI Conference if you’re open to it.
Alexander Lerchner spoke this year and was great.
I’ll reach out on email if you have a preferred one, it was unclear on your site which email you prefer.
I do want to pick a nit in this one,
> and the only lab where
I do think Allen Institution for AI (AI2) is has coverage across most of these domains ( albeit their frontier isn't nearly so far out, hopefully the $152m NSF awarded them + Nvidia is a fruitful partnership there).
For example, robotics: MolmoBot, MolmoSpaces, MolmoAct, https://allenai.org/embodied-ai
Today, humans convert their labor to capital. Capital holders need labor (humans) to acquire more capital. When the price of inference for these robots becomes less than the price of labor then capital holders don’t need labor.
Obviously, AI impacts non-manual labor too, but a significant portion of the world population does manual labor.
Ignoring quibbling about inference not being the only cost and other issues and just accepting the proposed end state: this is very good if capital is effectively democratized, and apocalyptically bad if it remains highly concentrated in a narrow class.
Could you sleep easily knowing that you have one in your house? What if you oppose the political views of its creators?
This is a complete non-starter for already-built apartment blocks, terraced homes, and even most semi-detached. It's an interesting but costly solution for new detached homes.
Humanoids are a useful form factor because the already-built human world is, definitionally, built for humanoids.
Ultimately I shouldn't have to trust my robots. If my roomba or my dishwasher go haywire they won't pinch my finger off. They physically can't listen to me or spy on me. These are good robots.
My bet is that the final robotic revolution will use genetically modified human/animal bodies with replaced brains. You'll have to stretch your ethics a bit, but if you grow a bear genetically modified in a way that it has no consciousness or thought, it'll make a much better construction worker than any humanoid robot. You'll just need to wire it up with neuralink and then control it via LLM. Fast animals can be used to deliver packages, and giraffes for warehouses.
Not to say we won't get humanoid robots eventually, but I think there's probably some low hanging fruit for people to make some other kinds of solutions. Specialized robots for industrial environment, well-thought out appliances for the home.
It would be a bit surprising if the progression was Roomba -> humanoid robot.
Seems to me you'd do better if you built the robots with robots that were specialized in building robots.
So the robots will need to be weak, so even weak people can overpower it. But then it loses a ton of it's most promising abilities.
I don't mean this like a "rogue robot" situation. I mean it like the robot gets confused, or someone sees a walking $10M lawsuit in their home.
So at some point you have to trust that the tech is safe. Both in terms of "robot won't go off the rails", but also in terms of hostile actors can't remotely take over your robot while you sleep.
In terms of sleeping, personally, I would like for the law to mandate that robots must have a physical off switch, in a very visible location, that physically disconnects power. The switch should be illuminated while in the ON position. What makes me a bit pessimistic there is that we don't even have laws to mandate webcam indicator lights (e.g. a very tiny red LED) must be ON in hardware.
Machine vision should definitely be handled by ML, but motion actuation should be relegated in the realm of traditional PID style linear/nonlinear control. Again, the tech is cool, but the practical usefulness of using a full LLM as a controller will probably run into hardware limitations.
On PID: the field has been stuck trying to do analytical/optimization-based control for decades, and end-to-end control has shown incredible performance (e.g., SoTA cost of transport in legged locomotion) and robustness (e.g., not falling over when stepping on a pile of leaves) - while being far scalable (in terms of how fast it is to get a new robot up and running). Which is not to say it's perfect but it seems like it's a step in the right direction.
Better yet, companies like Physical Intelligence are doing good with hierarchical ("fast-slow") architecture to address both the intelligence and latency fronts.
So, yeah, for complex reasoning and sensory processing, LLMs are the correct choice, and Gemini is especially strong at spatial reasoning over ChatGPT/Claude. But for actual motion/actuation? LLMs are the wrong tool for the job, probably easier to have LLMs program a reusable workflow in a script for repeated tasks instead of invoking LLMs after the first time.
Presumably that company could use the Gemini Robotics product to eliminate the remote controller.
Sure Tau is teleoperated, but teleoperating an agile movement like that is actually really hard to get right and still involves AI to keep the robot balanced. Tau is a lot closer to real deployment than Google, and when it performs useful tasks it is simultaneously collecting the data to eventually automate those tasks.
I do think robotics would come up with more safety mechanisms (provably safe motion planning etc) just because the risk is a lot more serious than LLMs spitting half-truths
Nvidia also releases their Cosmos series models.
That makes them extremely easy to sell to industry. This unlocks, in principle, a large chunk of the potential untapped industrial automation market that still relies on human labor because the ROI of redesigning production lines didn't make sense.
More realistically it seems that llms in several years could help dramatically decrease costs of automation and make it available for more industries
It's easier to redesign the work to be robot-friendly than to deploy a humanoid robot and have it actually work.
Humanoid startup execs just don't talk about that.
That's the most Skynet thing ever.
If the robots stop when humans are too close, wouldn't that mean that robots for close interaction or handling of humans need a whole other level of control?
It's certainly not there yet for anything practical, there's also certain bits and structures that don't have accurate names during construction, and it is important to keep that in mind - so a robot is unlikely to understand what it means to say "put the left bit of this box onto this right bit" due to ambiguity, a human would understand that
Plus we have no good reliable accuracy testing data in most cases (most tests occur on a few demos, but that isn't a good representation of how must things work), popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, most companies and researchers have their own benchmarks here.
Plus companies lie alot, and do very dangerous things in thier videos, I.e. these robots should not be standing very close to humans, because of being dangerous.
There are also legitimate concerns of misuse of these robots that need to be accounted for, misuse does not have to be warfare, but can be as simple as confusing it while it is cutting tomatoes with a knife.
Turning doorknob is easy, and fail recovery is also being worked on, but we don't have reliable statistics anywhere on that. The hard part is on practical things, as in when placing bricks or attaching a part during manufacturing it needs to ensure that it is aligning everything correctly....and that's hard, while it is impressive, it is very irresponsible to keep humanoids at home (people are irresponsible when untrained), for example, lawnmowers injure about 6400 people a year...and that is not an everything machine.
Humanoids in general are...not appealing in specific, due to maintainable of joints, complexity, but robot arms in particular, expecially on wheels (check mobile aloha), are likely to be able to do tasks such as clean up in hotels, after a guest had left, or replace some cooks in restaurants (if their work is consistent)
I did professionally few prototypes with robots and progress is real yet very far from what the average customer would find reliably useful in menial tasks.
FWIW I do think https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... remains relevant, namely dexterity is also a hardware problem, grippers aren't hands. They even clarify "multi-finger dexterous manipulation remains challenging." and those aren't even fingers with a lot of sensors.
Humanoids are far from being remotely useful in real life scenarios, yet.
Fun fact: most (if bot all) qugv can’t go reverse on a stairway.
It would be bad to remove demand for humans from the economy, of course. Humans have inherent moral value, and so it's good that our current system gives them economic value as well. But there's more than one way to achieve that end, and the massive quantity on the good side of the scale suggests it may be worth investigating the others instead of opposing the advancement outright.
I question the premise that humanoid robots are "around the corner". I suspect this will turn out more like self-driving cars which are still a very slow burn.
I’m not even sure if you are joking there or haven’t thought your proposal through. Sure giraffes are tall, but they can barelly lift any weight. What use would a robotically controlled giraffe be in a warehouse?
> There has been no innovation in robotic actuators since Honda's Asimo.
I very much doubt this. If nothing else the MIT Cheetah’s actuators are a whole different ballgame compared to asimo’s actuators. (Backdriveability, variable stiffness) And then there is a lot of interesting work being done with combining elastic elements with the actuators.
I do agree though the humanoid form is a dead end for robots. Just build giant cubes that process inputs and give outputs, like a dishwasher. Why wash dishes with meat wand tentacles or try to recreate meat wand tentacles when you can accomplish the job in a wholly different way with far greater efficiency...?
Where's the clothes foldeing cube? Analogous to the clothes washer and clothes dryer.... the clothes folder...
Why stop at dishwashing...? Sell an entire integrated robotic kitchen.
It's interesting to learn actuators are "behind". I kept seeing cool stuff in the 3D printing space and thought there's a lot of progress. I'd love to learn more.
Also the process that’s running at 16MHz is not the same as a full VLA. VLA is much more expressive. You think the processor that is running the VLA system is running at 10Hz or some GHz?
And if all goes bad, imagine a world in which you and your family are homeless and starve to death.
Luckily the politicians and business leaders in place today who are going to be responsible for navigating us to one of these outcomes are the adults in the room, very ethical, even keeled and not the least bit corrupt. So... we should be fine! /s
I am imagining a world somewhere between the movie Elysium and Oblivion.
Initially, sure. But the price will come down in time, just like with any other technology in history.
But maybe they'd rather deal with a robot than a human?
That aside: if I'm lucky enough to find a person who's good and reliable, they might move away, switch jobs etc.
It has all the headaches that come with hiring and managing someone, because, well, it is exactly that... If I don't want to be a manager at work (been there done that, happy to let others do it and get the raise that comes along with it), I sure as heck don't want to do it at home.
More seriously:
- I wouldn't feel obligated to 'pre-clean' anything. Who wants to come across as a thoughtless/careless slob whose personal habits amount to borderline-abusive demands on the cleaning staff? Not me... and where does that line get drawn, anyway?
- I wouldn't worry about stuff being picked up and misplaced
- I have never had problems with maids stealing stuff, but I know others have
- Cleaning can be initiated or postponed as needed, with no dependencies on someone else's schedule
For example, you could store far more things out of easy human reach.
Well, as long as you don't mind the workers in the robot company's telemetrics department whacking it to your robot's video feed.
I don't want a person I barely know regularly going all over my apartment. What if they discover... nevermind.
A robot, on the other hand? Hell yeah!
In the humanoid robot scenario, you'll get a surveillance device with a built-in microphone and camera beaming every intimate detail of your space to the highest bidder. Instead of getting one person you don't know very well in your space, you'll get thousands.
I think house cleaning/chores require at least a "part-time" job's work of work for the average household (especially with children). We may be atypical, but my partner and I don't want to hire out a cleaning service and deal with the whole human component. But, I think we'd gladly pay 15k+ if it allowed us to take on the additional gainful workload. Even at a 20k price tag, I suspect we would likely have made up the difference in under a year.
Selling to businesses is much easier, 99% of businesses can afford 200K if it replaces 3 x 70K humans.
I'd gladly pay or finance a robot that could have dishes clean, laundry folded, and carpets vacuumed by the time our family got home.
You can pay over $500 for an automatic cat litter box. Robot vacuums can be cheap but run to $600 or so. Household appliances is a robust sector where people have a proven track record of spending a lot of money.
Interested. Which one?
If I can get a robot replace a gardener who I pay $170/mo for 2 visits per month, i probably would. But I suspect that would kill lawn
This is my dream for a house chore robot. If I could dump hampers of clothes into a receptacle and get stacks of folded clothes out, I’d happily pay $1k. Household of 5; my kids each produce 3-4 sets of clothes a day (sleep, school, sports, after school). I run wash in AM so dryer finishes before 3pm (pg&e ToU) and then I’m focused on other things for rest of day. More often than not, I get to bed to find a near full hamper of clothes dumped where I sleep and then have to sort/fold/deliver. Bonus points if robot can sort different items to different stacks/bundles.
I don’t need AI/robotics to save me time from having to think, research, or code… I need AI/robotics to save me time so I can think, research, or code.
Like cars, they'll likely purchase them through finance deals, with a monthly payment. Or perhaps even rent/lease them.
If the future home robots are any good, it saves you from buying a dishwasher and robot vacuum. It can replace a maid/cleaning service and gardener/lawn-mowing service. For anyone paying for those services, it pays for itself.
$1000 isn't that high. Probably about a third of American households have an appliance over $2000, which is still a massive market.
And talking of kids, these will have to be exceedingly safe, a 6ft machine falling on a child is probably a worry most households could do without.
They all have cars. Turns out you can finance things, and turn them into forever debts. Paying $30K to never have to do chores again, or maybe even cook again, and live in a clean neat environment is not only an amazing proposition that opens up a lot of free time for people, but will also have a lot of social pressure behind it. We have social pressure that causes people to buy >$1000 phones to virtually no marginal benefit over $20 phones.
They will be paid off at $500/mo over 10 years, and most renters might just use the one that came with their apartment. If they're more like $80K or $100K I could see having a problem selling them. But if they basically turned apartment buildings into hotels, they would be a bargain for landlords; just give them the keys.
So the lifestyle stayed inflated.
How big is your house? When we had a cleaning lady it was $70 a go. Under $2000 for every second week.
I think I remember maybe around $100 including laundry almost 20 years ago and that was a cash type arrangement, weekly including laundry (multi hour visit).
Wondering if you're closer to the southern border or something.
1. 80%+ of Americans can't afford house cleaning services. I'm a software engineer in silicon valley and even I clean my own house because I don't think I can afford it. My rents have gone up by more than 5K in the last year, my salary hasn't, so that's coming out of my house cleaning budget.
2. Even if you can afford 5K/year on it, the cleaning service is not an upfront commitment. You can bail anytime when you get laid off and clean your own house.
3. Consumers don't budget rationally. Businesses do.
But that means speed does matter after all. The renting company will need to clean as many flats as possible with a single robot in a single day.
Robots aren't a necessity, and for a price tag in the tens of thousands, most people will just mop their own floors and do their own laundry.
And (regardless of the costs involved) if I were going to pay money to not do this stuff myself, I'd rather pay a human to do it because I know there are humans out there that could use the work and I value them much more than I value Google and other corporations making even more obscene amounts of money selling future e-waste.
If Rosie was the only output she would never happen. Rosie might happen as a side benefit of replacing American workers but only for the rich people.
The vast majority which may be replaced would probably be better off burning down the factory rather than celebrate their upcoming domestic helper.