Dark Hacker News
new
|
best
|
ask
|
show
|
jobs
Tokenization Is a Problem for LLMs | Dark Hacker News
Tokenization Is a Problem for LLMs
(blog.dottxt.co)
2 points
by
remilouf
2 years ago
| 1 comment
remilouf
2 years ago
|
next
[−]
This article presents a way to make structured generation with LLMs much faster than standard generation, but what I find most interesting is how it highlights the issues that tokenization entails towards the end.