In summary, they forced the model to process data in ternary system and then build a custom FPGA chip to process the data more efficiently. Tested to be "comparable" to small models (3B), theoretically scale to 70B, unknown for SOTAs (>100B params).
We have always known custom chips are more efficient especially for tasks like these where it is basically approximating an analog process (i.e. the brain). What is impressive is how fast it is prgressing. These 3B params models would demolish GPT2 which was, what, 4-5 years old? And they would be pure scifi tech 10 years ago.
Now they can run on your phone.
A machine, running locally on your phone, that can listen and respond to anything a human may say. Who could have confidently claim this 10 years ago?
Combined with the earlier paper this year that claimed LLMs work fine (and faster) with trinary numbers (rather than floats? or long ints?) — the idea of running a quick LLM local is looking better and better.
Some more discussion a few weeks ago: https://news.ycombinator.com/item?id=40620955
The whole point of AI was to sell premium GEMMs and come up with funky low precision accelerators.