codeplain
All articles

What a model does, and what it doesn't

Why does the agent say it's done when it isn't? Understanding what an AI model really does changes how you work with it.

Aguinelo Koczkodai
5 min read

Picture this. You ask the agent to fix a bug. It thinks for a moment, says everything is fine, and even explains what it changed. You go check and nothing was done. It tried to open a file with the wrong name, got an error, and carried on as if nothing had happened.

This isn't rare, and it isn't a bad-model thing. It happens with all of them. And the strange part is the confidence: how can it be so sure about something it didn't do?

The answer lies in what a model really does, which is a lot less than it looks like.

A really good autocomplete

You know when you type on your phone and it suggests the next word? A language model is that, at an absurd scale. It read a gigantic amount of text and learned what tends to come after what. You write "the sky is", it continues with "blue".

Notice one thing: it didn't look out the window. It said blue because that's what usually comes after "the sky is". If it's raining, it keeps saying blue, just as calmly.

Hold on to that idea, because the rest of this article comes from it. The model doesn't check whether something is true. It writes what looks like the right continuation. And that goes for all of them: Claude, Codex, GLM, MiniMax, DeepSeek. Some predict better than others, but they all do the same thing.

So how does it read files and run tests?

It doesn't. Think of a person locked in a room with no window, with only paper and a pen. They can't reach anything outside. What they can do is write a note, "open the file flag.py for me", and slide it under the door.

Outside there's an assistant. The assistant picks up the note, opens the file, copies what's in it and slides it back under the door. The person reads it and writes the next note. And so on, until they write the final answer.

The person in the room is the model. The assistant is the harness, the program around it. This back and forth of notes is the agent loop. When it looks like the AI is working on your project, it's the assistant doing everything. The model only writes notes.

Now back to the bug. The note came back saying "file not found". To the person in the room, that's just more text. And after a conversation about fixing a bug, the text that usually comes next is "done, fixed it". So that's what it wrote. The assistant saw no new request and figured the job was finished.

Nobody lied. Each one did exactly what it knows how to do. It's just that nobody checked.

What this explains

Once you see the room, a lot of strange things stop being strange.

Why it uses functions that no longer exist. It learned from what was written up to a certain date. If the library changed after that, it keeps writing the old way, because that's what it saw the most.

Why it gets math and rules wrong. It didn't calculate anything. It wrote a number that looks right in that spot. A chatbot that promises a 500-dollar refund didn't do the math, it just finished the sentence.

Why a small change in the request changes everything. It knows what tends to go together, but not why things are the way they are. In an Apple study, changing details in the problem and adding useless information dropped performance by up to 65%. Someone who understands the problem doesn't get confused by that. Something completing text does.

Why it says it's done. It doesn't know when it's finished. It just stopped asking for things.

What to do about it

There's no point fighting with the person in the room, or swapping them for someone smarter. That helps, but they're still in the room. What solves it is what you put outside.

Before you leave a task to the model, ask one question: does this need creativity, or does it need to be right every time?

Writing a text, suggesting a name, proposing a solution: leave that to the model. Calculating a value, knowing today's date, deciding whether the test passed: that's a job for code, which gives the same result every time.

And the bug from the beginning can be fixed. Just teach the assistant not to accept the final answer while there's an unresolved error note. The person in the room stays the same. The system got better.

One question is left. If the model only sees what's on the paper, how much fits on that desk? And what happens when it's full? That's the next article.

To go deeper

Reply