Ask HN: Do AI code tools work well with your company's codebase?

8 points by prasoonds 1 year ago | 4 comments

Hey HN, I'be been using Cursor a fair bit recently and I really enjoy it but it doesn't work too well when I ask it questions that use the whole codebase.

Normally, I'm working in a small part of the codebase and I can give really concrete instructions and the right context to Cursor. Here, it works really well.

But, when I ask it questions are are more generic (where's the code that does X/how can I do X?/Implement <functionality> by using X and Y), it often hallucinates or gives me wrong answers.

I can see that it tries to find the right context to send to the LLM. Sometimes, it does find the right context, other times it doesn't. Even when it does find the right context, I'm guessing it's just too much context that gets sent to the LLM so it just ends up hallucinating.

Have you had this same problem with whichever AI coding tool you use? I'm wondering if the problem is specific to the legacy + large codebase I'm working with or is it a more general thing with unseen code that the LLM hasn't seen in it's training data.

dgosling56 1 year ago |

It's not that great at answering questions specific to my company's code base - like where do I find another API method that does Y. However, we have a lot of guides internally which are usually scattered around in github readmes and usually those are what I fallback to when trying to find an answer. These github readmes and be quite long though and it can feel like a lot of manual work to scan through them, that's where I'll usually use this extension Rocky AI to quickly ask a question given the context of a webpage (helps avoid reading through the entire thing manually). After quickly browsing 2-3 and I'll have what I need.

https://chromewebstore.google.com/detail/rocky-ai/fdjoklehji...

skeptrune 1 year ago |

They don't work well at all for us.

We have a relatively large Rust codebase and I would describe the LLM's as actively harmful at making agentic-style, self-piloted changes.

Very pointed context management with split screens and explicit inclusion of relevant files in the context is sometimes useful though.

My biggest gripe is that AI can't just generate a new SQL migration and CRUD routes for an additional resource we want to add without issue. I always have to punt and end up doing it myself.

prasoonds 1 year ago | |

I've heard Rust is especially hard for LLMs, what with the need to thinking deeply about variable ownership and lifetimes. I wonder if it will work better if you try o1-mini (as long as you're still providing the right context!)

And I would've thought that AI would be good at creating new routes (wrap this function in a POST route) - certainly I've done this with Python and TS and it's been fine. Maybe it's a Rust specific issue? Likewise, SQL migrations I would've expected to also just work, especially if your schema change is small! Interesting that it doesn't!

Which tool do you use? Cursor or something else?

prasoonds 1 year ago |

Also to add some more context:

- I've found LLM codegen working very well with standard React + TS code

- It sucks when using less knows languages or less popular frameworks (I tried reverse engineering github copilot lua code from the neovim extension in one instance and it really didn't work well)

I'd be curious to hear people's experience here.