Stop using JSON for LLM structured output

2 points by 44za12 257 days ago | 1 comment

44za12 257 days ago |

For simple extraction tasks, a delimiter-separated string uses 11 tokens vs 35 for JSON. Output tokens are the latency bottleneck.

44za12 257 days ago |

For simple extraction tasks, a delimiter-separated string uses 11 tokens vs 35 for JSON. Output tokens are the latency bottleneck.