Anyone working on LLM tools for enhancing data quality?

5 points by cstanley 2 years ago | 7 comments

Big problem, let's break it down...

1. Data issue identification 2. Solution and implementation

Most issues are discovered in the data warehouse. Entity matching customer data across different systems, some business process results in duplicate, or null data. I know there are existing, non-LLM, products that do this. I'm curious to compare those with new LLM first products.

On solution/implementation. Ideally you're able to fix this in the source system, either in your SaaS tool or in the way you write production data. You can also fix this in the data warehouse, munging/ETL'ing the data. Seems like LLMs could help to 1) identify and recommend a change in an external system, 2) submit a PR to solve this in the data warehouse.

Anyone know anyone working on these problems?

hxypqr 2 years ago |

Obviously, the cost of using LLM to solve this problem is relatively high. At most, it can only treat LLM as one of the signals instead of processing all the information. Combining LLM with tree models or graph models and adding NER technology may achieve an approximate solution to 2. Can you really trust the answers provided by LLM to be used in a production environment?

johannesboyne 2 years ago | |

The combination is truly key, how we do it:

- Combining LLMs, gradient boosting, and multiple statistical/heuristic based approaches - For LLMs: rely on (quite) large context length + fine-tuning - The (ML) models are used to extract mathematically provable quality checks

latentpot 2 years ago |

1 is already a solved problem. My employer had originally put a ML based system to find DQ issues (already working) and is looking at pocs to add LLMs in the model mix. Hearsay is that our lakehouse vendor will have their own solution to this question via a acquisition.

2 is interesting, possible to do via LLM but I worry about data privacy and hallucinations making data more believable but not real.

johannesboyne 2 years ago | |

reg. 1) Which one has been used?

spdustin 2 years ago |

I'm using LLMs to help build dbt-flavored Great Expectations tests. Pretty low-hanging fruit, but it's a start.

johannesboyne 2 years ago |

We (https://dqc.ai) are doing something in the space, yes. Next to a mixture of ML, and heuristic based approaches, plus link & integrations into source systems. Happy to talk about it, feel free to reach out.