Any text-to-SQL benchmark should address difficulties of real-world data stores

Any text-to-SQL benchmark should address difficulties of real-world data stores(cacm.acm.org)

26 points by shenli3514 3 hours ago | 4 comments

Interesting... at my current job, my team and I are solving this problem of how to make sure LLM understand our SQL data warehouse to answer analytical questions for clients. We have a 20-year old database with rotting schema as the blog post describes. So we had to rebuild the database in a way that is well structured and governed. Once that hard work is done, we slap dbt models on top of business metrics with model YAML files (and some common MD files) carrying a lot of semantic and metadata info for them.

Then our software engineering team ingest the dbt models (we have to tactically create dbt models; that is, always think "what would make LLM hallucinate less" as we are implementing them) and info from the semantic layer to build context for the LLM, and use that to answer analytical questions. So far, it's been promising. The accuracy isn't zero like the blog's author suggested though. We have built like 30 metrics in dbt and semantic layer in the last quarter, and asked the research and analytics teams to do internal testing on the LLM app. I will find out how accurate this approach is from the feedback soon.

fivetenpen 2 hours ago |

The biggest issue with business users (who don’t know SQL) writing SQL with an LLM is that there is no one to validate that query and now that business user will treat that LLM response as canon to share in meetings, presentations, and with clients. The LLM may have forgotten a filter, used the wrong definition of revenue, or misunderstood the user’s intent by writing a query that answers their question in the most literal way.

That is the crux of the issue. No amount of semantic layers and context will help this until an LLM can read the user’s mind to remove ambiguity in the prompt.

I see most of the benefits of LLMs to be used by analysts who know SQL to work more productivly.

mmasu 27 minutes ago | |

also, the same metric can be used in different ways and for different purposes across a business, so you might need different validation rules and definitions.

Maybe one separate semantics validation layer could help, but costs 2x or possibly Nx if you need to recover and turn a wrong query into a correct one

semiquaver 2 hours ago |

  > It is widely known that an LLM can only find data it has seen before.

What does this mean? It’s obviously not literally true, so what is the author trying to convey?

m12k 2 hours ago |

Friend of mine built a startup around this, allowing non-techies to "query their database using natural language": https://www.blazesql.com/ Not sure how he achieved it (if TFA is to be believed), but it's my impression that his query generation and results are fairly robust.