After using code LLMs' for a bit, I wonder whether there is significant productivity gain in using them. I found that its limited for me.

Main reason: a) to find the right code, I have to keep sampling it, and b) it doesn't seem to be able to solve larger / more complex problems that I actually find more need for.

I found some interesting research on combining planning-algorithms for complex problems, and some ideas on guiding the LLM's decoding process towards correctness by optimizing it via reward functions and reducing the search space. I've detailed and summarised the main points in the post above.

Questions:

1) Do you find code LLM's really useful? Please share some stories / examples where they help vs they didn't. I'm trying to form a better understanding they are just fancy, vs actually productive and useful to most

2) Any other research ideas being pursued in this field ? / what are you trying ?