Show HN: LLM Based Spark Profiler

Show HN: LLM Based Spark Profiler(datasre.ai)

27 points by ambrood 1 year ago | 5 comments

Hey HN,

Spark event logs run into 100s of MBs and offer a wealth of insight into your workloads but making sense of them has always been quite a bit prohibitive. We’ve recently built a lightweight tool that automatically parses Spark event logs and surfaces targeted insights to help you optimize your data jobs.

Whether you’re chasing down a bottleneck or balancing performance vs. cost, the profiler got you covered with real-time configuration recommendations, data skew analysis, and more.

Curious how it works in action? Check out this quick Loom video for a walk-through: https://www.loom.com/share/07348eb54f6b440da93f96753937792a?...

We’d love your feedback — check it out at https://app.datasre.ai and let us know what you think!

vector_spaces 1 year ago |

Maybe you mentioned it in your demo and I missed it, but how does this differ pasting the log messages to ChatGPT / Claude / another LLM? Is it mainly that yours can iterate over a large logfile without blowing up the context window?

Does it suffer from the same issue as other LLMs, where it will always identify potential optimizations or improvements even if none are truly needed?

ambrood 1 year ago | |

> Maybe you mentioned it in your demo and I missed it, but how does this differ pasting the log messages to ChatGPT / Claude / another LLM? Is it mainly that yours can iterate over a large logfile without blowing up the context window?

We do quite a bit of aggregation over the log file, and generate summary stats and choose what bits to stuff in the LLM. Plan to support more platforms than just spark.

> Does it suffer from the same issue as other LLMs, where it will always identify potential optimizations or improvements even if none are truly needed?

Funnily enough, instructing sonnet-3.7 to not suggest unnecessary optimisations seems to have done the trick!

emgeee 1 year ago |

fellow co-founder here! One fun thing about this project is the entire frontend was vibe-coded using Bolt in a few days.

skeptrune 1 year ago | |

Very awesome. Not having to burn time on a UI that looks and feels nice is a huge win.

ztratar 1 year ago |

Also curious how the agent works?