Stop using JSON for LLM structured output

2 points by 44za12 114 days ago | 1 comment

44za12 114 days ago |

For simple extraction tasks, a delimiter-separated string uses 11 tokens vs 35 for JSON. Output tokens are the latency bottleneck.

44za12 114 days ago |

For simple extraction tasks, a delimiter-separated string uses 11 tokens vs 35 for JSON. Output tokens are the latency bottleneck.