To actually engage more with the substance of TFA.. very refreshing to see someone bringing numbers. To me this shows we (still) need something somewhere between the numbers and the anecdata. It's annoying to hear claims of "1000x productivity" or claims of negative/neutral productivity without any extra context whatsoever. So you brought data? Great! But still no context. Boring CRUD? Complex UI? A rewrite/port of legacy? What industry, language, how many human collaborators, and what baseline for SLOC??

We need to get rigorous about this stuff and actually aim for a decent framework which can answer "Are LLMs a value add for this project? How much value? How much cost?"

Such a thing might be information-theoretical, complexity-based, or counting integration touchpoints / info boundaries / sources of ground-truth? We could even try to implement that framework with LLMs and probably should! But the default answer of "Yes definitely, always useful, just token-maxx it since LLMs are the future" is (still) only marketing, not engineering

This is like complaining a student isn't as productive as a senior engineering.

I think we as an industry haven't even graduated to junior level when it comes to figuring our how to use AI to improve things.