What happens behind the scenes when we change effort for same LLM models? What really happens behind the scenes when we select high effort vs medium effort for the same LLM models? |
What happens behind the scenes when we change effort for same LLM models? What really happens behind the scenes when we select high effort vs medium effort for the same LLM models? |
---
Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
> When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.
---
Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token":
https://magazine.sebastianraschka.com/p/controlling-reasonin...
> During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.
Now how true that is, I don't know. You could reasonably implement this locally with an inference provider, but I don't know any that do, or I haven't used them at least.
I may also have the technical detail incorrect, but it did come down to essentially "continuing inference beyond a natural termination."
https://arxiv.org/pdf/2505.09388
> Thinking Budget. An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model’s thinking process. Specifically, when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: “Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n”.
^-^
One is the sous chef , who knows recipes and cooks fast, handle most dishes as quick as possible. He does not think much. That is your medium effort llm in charge.
One is the head chef, who is inspecting, pause, consider edge case. Slower but catches what sous chef missed. That is your high effort llm in work.
models dont seem to care if you cut them off.