LLMs learn what programmers create, not how programmers work

41 points by noemit 55 days ago | 22 comments

I ran an experiment to see if CLI actually was the most intuitive format for tool calling. (As claimed by a ex-Manus AI Backend Engineer) I gave my model random scenarios and a single tool "run" - i told it that it worked like a CLI. I told it to guess commands.

it guessed great commands, but it formatted it always with a colon up front, like :help :browser :search :curl

It was trained on how terminals look, not what you actually type (you don't type the ":")

I have since updated my code in my agent tool to stop fighting against this intuition.

LLMs they learn what commands look like in documentation/artifacts, not what the human actually typed on the keyboard.

Seems so obvious. This is why you have to test your LLM and see how it naturally works, so you don't have to fight it with your system prompt.

This is Kimi K2.5 Btw.

mpalmer 55 days ago |

The novice came to the master. "I have figured it out, the rules for how LLMs understand CLIs. It gives the right commands, but adds colons. It was trained on the visual shape of terminals, not keystrokes."

"Clear the session," the master said. "Run the same prompt again."

The novice pressed return. The model output: `ls -R /tmp`

"The colons are gone," the novice said. "But my theory explained them perfectly."

"You built a cage for a cloud," the master said. "Do not mistake a single roll of the dice for the rulebook."

noemit 55 days ago | |

I ran tests of 100 attempts with different prompt/scenario combinations. Each "attempt"/theory had 3 different system prompts wordings. Most of the prompts did not mention a colon, but it kept appearing. When I added negative instructions against using a colon, the quality went down (most of the tool calls were malformed, one common issue was markdown ticks in front) It was only when my system prompt acted like colons were normal that I kept getting 100/100 perfect expected tool calls. I ranked my system prompts by which returned the most consistent commands.

shomp 55 days ago |

Great observation. The brain of a programmer is still a "black box" to the feed-forward network of nodes . But in theory, if you pumped a lot of the live-coding videos from something like youtube into the process, you could get a bit of that "what's your approach"-erism to bleed into the model. There might not be enough material there to truly "train it to think" but it would be interesting to try and "fill the gaps" of black-box-ed-ness in the LLM with supplemental "here was the process that got us there" video feeds. The next natural move might actually be recording thousands of hours of footage of developers working with the LLMs directly like in Cursor or another IDE that has LLM live-pair-programming , maybe calling it "pair programming" is generous , but it might be a reasonable foray into teaching the next generation of LLMs the "thought process" behind things. In reality you'd be teaching it which files to inspect, which windows to open/close, which tools to switch to and focus on. And while it might be imperfect, it might just be enough.

agent_anuj 49 days ago |

This is coming straight from my experience last week. I actually tried to test this. took 30 days of my claude code sessions, about 32k conversation turns across 21 sessions and 10 projects. classified every user message - corrections, feedback, decisions, reframes. extracted about 3200 high signal training pairs. I put a lot of emphasis on my explicit corrections where I told the AI it was wrong and what the right answer was and WHY. fine tuned qwen 4B on it with qlora. the model learned my voice perfectly - during training it would say things like 'no. fix the query. you're doing 3 joins when you only need user_id' which is exactly how I talk. but thats the problem - it learned to parrot my phrasing without understanding why I made those corrections. it memorized the what, the artifact, but completely missed the how - the reasoning process that led to the correction. Title is exactly right.

KalskiTheDan 52 days ago |

I ran into this building data pipelines for LLMs. Kept feeding structured JSON with numbers and the model would do bizarre arithmetic on them.

The moment I shifted to computing the analysis upstream and describing results in plain english, the outputs were much more coherent.

The model doesn't know what numbers/spreadsheets of data mean inherently, as in... the LLM inherently does not compute a math equation/formula within itself. Rather, an LLM would calls/creates calculation code on the side to then read off the results. And still with this side-work, the code/calculator created would give contextual wordings to the numbers; highest, average, x% growth, consolidating at y% rate, etc.

paulwelty 53 days ago |

You found an interesting example of the fallacy of following and defining rules You can't define a system with only rules. Eventually, you run out of rules on how to apply the rules. Eventually, you need judgment and interpretation.

alexchansg 50 days ago |

Great insight. We noticed something similar when building our agent tool — LLMs would output git commit -m "..." with the quotes already formatted, even when we just wanted the message body. They're mimicking documentation examples, not actual command entry.

Ended up just accepting it and adjusting our parser. Fighting the model's training is a losing battle.

mememememememo 52 days ago |

Not sure it is true LLMs don't see code or cli commands directly in their training. They go through reinforcement learning and they could easily be trained on a command line. People are paid to give human feedback. See https://huyenchip.com/2023/05/02/rlhf.html

Areena_28 55 days ago |

I know even we hit the same thing building internal security tooling. our model kept formatting output like documentation, not like how we would or any person in place of us would read in a terminal at 2am during an incident.

I am a bit curious, did you find this behavior consistent across models or is it more pronounced with certain ones?

noemit 55 days ago | |

I ran into it while building - I should have tested different temps too - I was just trying to get cli style tool calls to be more reliable

Areena_28 52 days ago | | |

yeah temperature is probably worth a run, we noticed even small adjustments changed how the model interpreted formatting expectations quite a bit.

stuaxo 55 days ago | |

Literate programming is about to become mainstream in the funniest way possible.

Areena_28 52 days ago | | |

oh yesss, except literate programming was still the human explaining intent to other humans. this is more like the human explaining intent to a machine that then explains it back to other humans. hahaha this is actually funny.

acters 55 days ago |

Instead of telling the LLM that "run"works like a cli, maybe just tell the LLM that "run" will execute sh/bash/zsh/etc scripts?

noemit 55 days ago | |

I tried over 20 variations of different system prompts. Once I changed my tool to expect the colon, it also felt like it was running/calling tools faster, but I need to do a larger test to be sure.

seertaak 55 days ago |

Is that really true? I would have expected by now that AI companies nowadays are doing RL on git histories, not just on the HEAD.

noemit 55 days ago | |

I also expected this. Please run some experiments and maybe other models are different

muzani 55 days ago | |

Claude definitely does

tucaz 51 days ago |

What changed after this discovery? What was better or worse and how better or worse?

infosecphoenix 54 days ago |

not just for coding, for any profession, LLMs only learned what people created but not how professionals work.

shivang2607 52 days ago |

And then they say AGI is coming

Art9681 55 days ago |

Is "how programmers work" a useful and provable metric? No? Then it belongs in philosophy discussions. How you work and how I work is different. Your work may have ended up in the LLM training and my work did not. Or vice versa.

Can you objectively analyze how VSCode adapts to your way of working without our interference?

Did you test your theory with the actual frontier LLMs (which Kimi K2.5 is not BTW?)