Small AI Models Gain Traction In places with unreliable networks

Small AI Models Gain Traction In places with unreliable networks(spectrum.ieee.org)

25 points by sscaryterry 3 hours ago | 5 comments

tim-fan 1 hour ago |

Is anyone making LLM-in-a-box for emergency supply kits yet?

I feel that would be handy in all sorts of situations when networks are down.

Terr_ 8 minutes ago | |

I imagine that for most "emergency" scenarios, it'll be a lot cheaper and more-robust to focus on dumber devices that store large amounts of normal reference material, as opposed to a system with the battery/CPU/memory/etc. to run an LLM.

If someone needs something like a medical flowchart for first-aid, that's probably something better to generate in-advance.

SwellJoe 21 minutes ago | |

This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/

But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.

cdnsteve 42 minutes ago | |

Can you expand what you mean?

wahnfrieden 31 minutes ago | | |

They want to ask the iOS Foundation model (frontier on device intelligence for something small) for instance about emergency procedures and life-saving info. I wouldn’t trust that model with much at all though. More likely to find what you need from miniature survival guides.

bombcar 1 hour ago |

99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.

enoint 1 hour ago |

Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.