Show HN: Alerting in realtime RAG: spot changes to LLM answers, using few tokens

Show HN: Alerting in realtime RAG: spot changes to LLM answers, using few tokens(github.com)

8 points by janchorowski 2 years ago | 5 comments

Hi I am Jan, CTO @ Pathway.

A use case we have been working on with LLMs is to let people know when an answer to their query changes due to revisions of source documents. Obviously, we want to avoid periodically re-computing all queries for the LLM.

Why I think it’s cool? - We don’t spin in a loop to repeat with the LLM. - Alerts are LLM-deduplicated - no spamming users with typo fixes - And the best - our framework, Pathway takes care of handling the updates, the example looks nearly like a regular, static RAG chatbot.

More context + GIF of how it works for Google Drive document alerts: https://pathway.com/developers/showcases/llm-alert-pathway

Happy to have your thoughts!

bobur_umurzokov 2 years ago |

Thanks for sharing, Jan!

This real-time alerting use case can be also useful in many other areas. I am thinking of fraud detection, customer support, medical diagnosis, and treatment, or in manufacturing to predict when equipment will fail and alert if maintenance is needed. Or even monitoring model performance when LLMs can occasionally produce unexpected or undesirable outputs.

anupsurendran 2 years ago |

Jan, can you explain briefly how the deduplicator checks if the new answer is significantly different? Is there code in the repository we can take a look at?

janchorowski 2 years ago | |

Sure: when a new response is produced because some source documents have changed we ask an LLM to compare the responses and tell if they are significantly different. Even a simplistic prompt, like the one used in the example would do:

    Are the two following responses deviating?
    Answer with Yes or No.

    First response: "{old}"

    Second response: "{new}"

(used in https://github.com/pathwaycom/llm-app/blob/69709a2cf58cdf6ea...)

pstorm 2 years ago | | |

Couldn't you just compare the similarity of the embeddings? I imagine that would work in the vast majority of cases and save a lot of LLM calls.