Structured Output with LangChain and Llamafile

Structured Output with LangChain and Llamafile(blog.brakmic.com)

46 points by brakmic 330 days ago | 20 comments

Hugsun 326 days ago |

The version of llama.cpp that Llamafile uses supports structured outputs. Don't waste your time with bloat like langchain.

Think about why langchain has dozens of adapters that are all targeting services that describe themselves as OAI compatible, Llamafile included.

I'd bet you could point some of them at Llamafile and get structured outputs.

Note that they can be made 100% reliable when done properly. They're not done properly in this article.

halyconWays 326 days ago | |

>Don't waste your time with bloat like langchain.

Amen. See also: "Langchain is Pointless" https://news.ycombinator.com/item?id=36645575

pcwelder 326 days ago |

```

try:

    answer = chain.invoke(question)

    # print(answer) # raw JSON output

    display_answer(answer)

except Exception as e:

    print(f"An error occurred: {e}")

    chain_no_parser = prompt | llm

    raw_output = chain_no_parser.invoke(question)

    print(f"Raw output:\n\n{raw_output}")

```

Wait, are you calling LLM again if parsing fails just to get what LLM has sent to you already?

The whole thing is not difficult to do if you directly call API without Lang chain, it'd also help you avoid such inefficiency.

moribunda 326 days ago | |

I don't get the langchain hate, but I agree that this "blog post" is bad.

Langchain has a way to return raw output, aside "with structured output": https://python.langchain.com/docs/how_to/structured_output/#...

It's pretty common to use a cheaper model to fix these errors to match the schema if it fails with a tool call.

crystal_revenge 326 days ago | | |

> It's pretty common to use a cheaper model to fix these errors to match the schema if it fails with a tool call.

This has not be true for a while.

For open models there's 0 need for these kind of hacks with libraries like Xgrammar and Outlines (and several others) both existing as a solution on their own and being used by a wide range of open source tools to ensure structured generation happens at the logit levels. There's no-need to add multiples to your inference cost, when in some cases (xgrammar) they can reduce inference cost.

For proprietary models more and more providers are using proper structured generation (i.e. constrained decoding) under-the-hood. Most notably OpenAI's current version of structure outputs makes use of logit based methods to guarantee the structure of the output.

dcreater 326 days ago |

People still use langchain?

anshumankmr 326 days ago | |

Its good for quickly developing something but for production, I do not think so.We used it for a RAG application I built last year with a client, ended up removing it piece by piece, and found our app responded faster.

But orgs think its some sort of flagbearer of LLMs.As I am interviewing for other roles now, HRs from other companies still ask for how many years of exp I have with Langchain and Agentic AI.

reedlaw 326 days ago |

The use case in the article is relatively simple. For more complex structures, BAML (https://www.boundaryml.com/) is a better option.

kristjansson 326 days ago |

It's right there. In the screenshot in the blog post. Grammar > 'JSON Schema + Convert'. That's what structured output is.

... it's going to be september forever, isn't it?