Ask HN: Can On-device AI solve projected energy crisis?

1 points by vkkhare 1 year ago | 3 comments

bigyabai 1 year ago |

No, on-device AI is not a replacement product for the high-consumption AI that is casuing the problem.

vkkhare 1 year ago | |

Isn't the inference cost of running these models at scale challenging? Currently it feels like small LLMs (1B-4B) are able to perform well for simpler agentic workfows. There are definitely some constraints but surely much easier than to pay for big clusters on cloud running for these tasks. I believe it distributes the cost more uniformly

bigyabai 1 year ago | | |

It is very likely that you consume less power running a 1B LLM on an Nvidia supercluster than you do trying to download and run the same model on a smartphone. I don't think people understand just how fast the server hardware is compared to what is in their pocket.

We'll see companies push for tiny on-device models as a novelty, but even the best of those aren't very good. I firmly believe that GPUs are going to stay relevant even as models scale down, since they're still the fastest and most power-efficient solution.