![\"The](\"https://quionie.com/studies/where-the-computer-runs.png\")
I work in tech. I use these tools every day, and part of my job is testing AI systems before they go anywhere near a client. So I spend my week watching how they break.
\nI'm pro-AI. Something bothers me more than the arguing does, though.
\nThe real problems and the fake problems get shouted at the same volume. Someone will spend an hour insisting ChatGPT is boiling the ocean, which isn't true. That same person will never mention that the overwhelming majority of deepfake video is intimate imagery of real people, made and circulated without their consent. Which is true, and measured.
\nThat's a bad trade. We're spending our attention in the wrong places.
\nSo I went and read the research.
\nProteins are the machines inside your body. They do everything. What a protein does depends entirely on the shape it folds into.
\nIf you know the shape, you can design a drug that fits it, the way a key fits a lock. If you don't, you're guessing.
\nFor fifty years, working out one protein's shape took a scientist years of lab work. Expensive equipment. Slow going.
\nIn 2020, an AI system called AlphaFold cracked it.
\nTwo years later the team published predicted shapes for over 200 million proteins, which is close to every protein anyone has ever found, and gave the whole thing away free.
\nMore than 3 million researchers in over 190 countries have used it. Over a million of them are in low- and middle-income countries, places that could never afford the lab equipment and now have the answers anyway. More than 30% of the research built on it is aimed at understanding disease.
\nIt won the Nobel Prize in Chemistry in 2024. That same year the Physics prize went to Hopfield and Hinton for the neural network foundations underneath it. Two AI Nobels in one year.
\nThis one already ran, on real people.
\nSweden did a proper randomized trial. Over 105,000 women getting routine mammograms. Half got the standard process, where two radiologists each read the scan. Half got a version where AI read it first and flagged anything suspicious.
\nAI-supported screening found 29% more cancers, with no increase in false alarms. It cut the radiologists' reading workload by 44%, which matters more than it sounds, because there aren't enough radiologists and there won't be.
\nThen the number that made me sit up. They followed the women afterward and counted interval cancers, the ones that surface between screenings because a screening missed them. Those are the dangerous ones. The AI group had 12% fewer of them, and the ones that did appear were less often the aggressive kind.
\nA randomized controlled trial with a hundred thousand real women, and it means some people are alive who wouldn't be.
\nWeather forecasting has run for decades on giant physics simulations on supercomputers. The European system, ENS, is considered the world's best.
\nDeepMind built a model called GenCast and tested it head to head across 1,320 measures. Wind, temperature, pressure, storm paths, at different distances out.
\nGenCast was more accurate on 97.2% of them, and on 99.8% at lead times beyond 36 hours. It's strongest where it counts most: extreme weather and the tracks of tropical cyclones. Knowing three days earlier where a hurricane is going gets measured in lives.
\nIt produces a 15-day forecast in about eight minutes on a single chip. The traditional method takes hours on a supercomputer. Published in Nature, with the numbers in the paper.
\nThis part surprised me.
\nThe assumption is that AI serves the people already at the top. The evidence points the other way.
\nIn a study of 5,172 customer support workers, average productivity rose 14%. For the newest and least skilled, it rose 34%. The most experienced agents gained almost nothing, and on quality they slipped slightly.
\nThe same pattern showed up in a writing study with 453 professionals and a consulting study with 758 people. Every time, the biggest gains went to whoever was struggling most.
\nMost technology widens the gap between the best and the rest. At the individual level, this one narrows it. If you care who benefits from a new tool, that's unusual and worth defending.
\nNow the part where I don't defend anything.
\nThis belongs at the top of every list and it's almost never there.
\nNon-consensual intimate imagery, NCII in the research and the legislation, is exactly what it sounds like: intimate images of a real, identifiable person, produced and shared without their consent. The 2019 Sensity analysis found that 96% of deepfake videos online fell into that category, and every one it examined depicted a woman. A 2023 industry study put the figure at 98%. Different methods, different years, same answer.
\nResearchers and lawmakers have moved toward treating NCII as a form of abuse rather than a category of content, and the language matters, because "content" is what you moderate and abuse is what you prosecute.
\nThen fraud. Voice cloning got good enough that a few seconds of recorded speech produces a convincing copy. Criminals call parents impersonating their child in distress. They call finance departments impersonating an executive.
\nThe law finally moved. The TAKE IT DOWN Act was signed in May 2025, making it a federal crime to publish this material, and since May 2026 platforms have had 48 hours to remove it once a victim files a valid report. The first conviction came in April 2026, when an Ohio man pleaded guilty to charges including cyberstalking and publishing digital forgeries, after using AI to target women and children. Around 30 states have passed laws specifically covering the AI-generated kind, on top of near-universal state laws on NCII generally.
\nWhere I land. This is the strongest argument anyone can make against how AI is currently being deployed, and I don't have a rebuttal. I don't think one exists. What I'd add is that this isn't a mysterious force. Specific applications were built for it. Specific platforms hosted the output. Those are companies making decisions, and decisions can be regulated and prosecuted, which is finally starting to happen.
\nStanford has been tracking this with payroll records from ADP covering millions of American workers, and the number keeps getting worse.
\nWhen they first published in August 2025, employment for 22-to-25-year-olds in the most AI-exposed jobs was down 13% relative to their peers in other fields. The August 2026 revision, using data through June, puts it at about 19%.
\nTwo things make it convincing. Older workers in the same jobs are doing fine, so it isn't the economy and it isn't those industries dying. And almost nobody is getting fired. The drop is overwhelmingly jobs that never got created. Companies quietly stopped opening junior roles.
\nWhere I land. No counterargument. If you're 23, applying for junior jobs, and it feels harder than it should, you're right, and the best data available agrees with you. Anyone telling you to just learn AI is skipping the part where somebody still has to hire you.
\nMIT looked at business AI projects and found roughly 95% produced no measurable financial return. S&P Global found 42% of companies abandoned most of their AI plans in 2025, up from 17% the year before.
\nAnd there's one study I keep coming back to. Sixteen experienced developers, real work, in code they knew well. Half with AI, half without. They predicted AI would make them 24% faster. They came out 19% slower. Afterward they still believed they'd been faster. They couldn't feel it.
\nWhere I land. That's real and I take it seriously. It also sits next to several careful studies showing solid gains, so "AI doesn't work" is too strong. What's true is that it works on some things and not others, and people are bad at telling which is which. MIT's own conclusion about the failures is not what you'd expect from a report about failure: the problem is how companies implement it, not the technology.
\nGlobally it's smaller than people think. All data centres on earth, not just AI ones, used about 1.5% of the world's electricity in 2024. AI is a fraction of that.
\nGrowth is fast, and the IEA expects data centre demand to roughly double by 2030. The part nobody mentions is that energy used per AI task is falling at what the IEA calls a rate unprecedented in the history of energy. Total use still climbs, because demand grows faster than efficiency improves. But the guilt about your individual questions doesn't survive the math.
\nWhere the worry is completely right is locally. These buildings cluster. The US alone accounts for about 45% of global data centre electricity use. In Ireland, data centres went from 5% of metered electricity in 2015 to 23% in 2025.
\nIf you live next to one, your grid is straining and your bill is climbing, that is real, and quoting a global average at you is a useless answer.
\nWhere I land. Two different arguments keep getting mashed together. Planetary carbon, no. Your local grid, absolutely yes.
\nThe famous study came from MIT's Media Lab. People wrote essays with ChatGPT, with a search engine, or with nothing, while researchers watched their brain activity. The ChatGPT group showed the least engagement and, oddly, couldn't quote essays they'd finished minutes earlier.
\nWorth taking seriously. It also needs its caveats, and the lead author gives them herself. It hadn't been peer reviewed. There were 54 people, and only 18 finished the last session. And Nataliya Kosmyna has said plainly that they did not find brain rot and did not measure anyone's intelligence.
\nThe finding nobody repeats is the useful one. People who did the thinking first and brought AI in afterward did better on memory and engagement. The order mattered more than the tool.
\nWhere I land. Legitimate concern, thin evidence. I'd want a lot more before panicking or dismissing. The order thing I took personally, because it matches my own experience exactly.
\nThese are the ones that don't survive checking, and they crowd out everything above.
\n"AI is destroying jobs." Across all ages, employment in AI-exposed occupations has barely moved, and overall employment kept growing. The entry-level problem is real. The broad collapse is not happening. The exaggeration actively hurts, too: "AI is killing jobs" is a slogan nobody can act on, while "entry-level hiring is down 19% and no company has a plan for training juniors" is a real problem with a real shape, and it's the true one.
\n"Every question you ask uses a bottle of water." It traces back to one estimate, about an older model, in specific buildings, and the number swings wildly depending on where the building sits and how it's cooled. It got flattened into a per-question rule the original work never claimed. It also aims the guilt at you. You not asking a question doesn't change where a data centre gets built or what powers it. Those are decisions made by companies and utility regulators, and personal guilt is an effective way to make sure nobody looks at them.
\n"AI just glues together stuff it copied." This is wrong about how the thing works, and it matters, because the honest complaint is different. There's no folder of images inside a model. What's inside is a huge pile of numbers that got nudged, over and over, until the model got good at predicting what comes next. Nothing was stored to copy from later. Now, was it fair to use all that work without asking or paying? That's a real question and courts are working on it. But "it's copy-paste" isn't that argument. It's a claim about the machinery, and the machinery doesn't work that way. Making the wrong argument loudly is how you lose one you'd otherwise win.
\n"MIT proved AI makes you stupid." Fifty-four people. Eighteen in the final session. Not peer reviewed. And the author on record saying they found no such thing. If you're going to cite science at me, and you should, cite it the way you'd want your own cited.
\n"It's just autocomplete, so it can't do anything real." The description is roughly right. These systems predict what comes next, over and over. The conclusion doesn't follow. The same family of technology mapped every known protein, beat the world's best weather system, and found cancers two radiologists missed. "It's pattern matching" and "it can't do anything important" are two different claims, and the second keeps getting proven wrong while people keep saying it. I think it survives because it's comforting. It lets you skip the harder question, which is what it means that something this simple keeps working this well.
\nPro-AI. Specifically:
\nThe wins are real, they already happened, and they're bigger than most people realize. Proteins. Cancer screening. Weather. And at work, the people it helps most are the ones with the least experience.
\nNCII is the worst thing on the list and belongs at the top, not buried under arguments about electricity.
\nThe entry-level jobs problem is real, getting worse, and deserves more attention than it gets.
\nMost of the rest is either exaggerated or aimed at the wrong target.
\nUnderneath all of it, the technology isn't really the variable. How it gets used is. Nobody is going to stop it advancing, there's too much money and national interest behind it. But how it ships, what's legal, who gets protected, who gets hired, those are all choices. Made by people. Over and over. Including by me, at my job, this week.
\nThat's why I'm optimistic. Not because nothing bad happens. Because most of the bad things on this list came from decisions, and decisions can be made differently.
\nI'm not going to tell you you're being dramatic. On the two biggest items you're right, and one of them got worse this month.
\nWhat I'd ask is that we argue about the specific thing.
\nYour strong version is much stronger than the version that circulates. "AI is bad" gets nobody anywhere. "Non-consensual intimate imagery is 96% of all deepfake video and platforms took years to act" is a claim that has already produced federal law and criminal convictions. So is "entry-level hiring is down 19%."
\nUse the strong version. I'll take it seriously, and so will the people who currently tune you out.
\nThis is what I think in August 2026, based on what's published today.
\nSeveral of these numbers moved while I was writing. The Stanford jobs figure went from 13% to 19% across revisions of one paper in a year.
\nSo I'll come back to this. If the productivity results stop holding up, if the developer slowdown study turns out to be the rule instead of the exception, if the pattern of AI helping beginners most flips, I'll write it down and say what it changed for me.
\nHolding a position no matter what the evidence says isn't conviction. It's just a personality.
\nAlphaFold: the 2024 Nobel Prize in Chemistry materials, the 2024 Nobel Prize in Physics for the neural network foundations, and Google DeepMind's five-year impact review for the usage figures.
\nBreast cancer screening: the MASAI trial, Lund University. Interval cancer, sensitivity and specificity02464-X/abstract) in The Lancet (2026), with earlier results in The Lancet Oncology (2023) and The Lancet Digital Health00267-X/fulltext) (2025).
\nWeather: Price et al., Probabilistic weather forecasting with machine learning, Nature (December 2024), and DeepMind's GenCast writeup.
\nWork: Brynjolfsson, Li and Raymond, Generative AI at Work, Quarterly Journal of Economics (2025); Noy and Zhang in Science; Dell'Acqua et al. at Harvard Business School and BCG; METR's developer trial; MIT NANDA's GenAI Divide report; S&P Global.
\nJobs: Brynjolfsson, Chandar and Chen, Canaries in the Coal Mine, Stanford Digital Economy Lab, August 2025 through the August 2026 revision.
\nNon-consensual intimate imagery: Sensity's 2019 State of Deepfakes analysis and later industry composition studies; the TAKE IT DOWN Act (Public Law 119-12, signed May 19, 2025); reporting on the first conviction (April 2026); state legislative trackers.
\nEnergy: International Energy Agency, Energy and AI; Ireland's Central Statistics Office on data centre metered consumption.
\nCognition: Kosmyna et al., Your Brain on ChatGPT, MIT Media Lab preprint, plus the author's own clarifications.
\nAnything wrong here is mine.
", "image": "https://quionie.com/og/what-ai-has-done-and-whats-wrong.png", "date_published": "2026-08-22T12:00:00.000Z", "date_modified": "2026-08-22T12:00:00.000Z", "tags": [ "AI benefits", "AI harms", "deepfakes", "non-consensual intimate imagery", "AI and jobs", "AI energy use", "AlphaFold", "AI in healthcare", "AI productivity" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/ai-agent-architectures", "url": "https://quionie.com/blog/ai-agent-architectures", "title": "AI Agent Architectures: Workflows, Subagents, and Multi-Agent Systems", "summary": "Workflow, agent, subagent, multi-agent system and harness, explained in plain English, plus what the failure data says about splitting work across agents.", "content_html": "Every product in that agent comparison called itself an agent. So does a chatbot with a search button. So does a system that runs for six hours and rewrites your codebase. The word covers all of it, which means it tells you nothing.
\nThere are real categories underneath. There's also an argument about the most-hyped one, where two credible teams published opposite advice a day apart and neither has backed down.
\nAnthropic published the clearest version of this in December 2024 and it has held up.
\nA workflow is a set of steps you wrote out ahead of time. The model does the thinking inside each step, but you decided what the steps are and what order they run in.
\nAn agent gets a goal and works out its own steps as it goes. Which tool to use, what to do next, when it's finished.
\nA recipe versus a cook. With a recipe you made every decision and the cook carries them out. With a cook you say "make dinner" and hand the decisions over.
\nBoth get called agentic systems. The only real difference is who decides what happens next: you, in code you wrote, or the model, while it's running.
\nThe trade runs both ways. Workflows are predictable and easy to fix when they break, and they fall over on anything you didn't think of. Agents handle what you didn't think of, and you give up knowing in advance what they'll do. Every extra turn a model takes on its own makes the job slower, costs more, and gives an early mistake another chance to spread into everything after it.
\nAnthropic's own advice is to use the simplest thing that passes your tests, which is often a workflow or even one well-equipped model call, and to save agents for jobs where you can't write the steps out in advance but you can still check whether it's getting somewhere.
\nMost reliable systems running in production today are workflows, not agents deciding everything for themselves. Worth knowing before you buy something sold on autonomy.
\nBefore any of the patterns there's what Anthropic calls the augmented LLM. That's a model with three things attached: it can look things up, it can use tools, and it can remember.
\nThat isn't an agent yet. It's the unit the rest of this is built out of.
\nAll five count as workflows, because in each one you wrote the order.
\nPrompt chaining. Break the job into steps that run one after another, with a check in code between each one. Step one drafts, step two checks it, step three formats it. Simple, and mistakes get caught early because you can stop it at any step.
\nRouting. Sort the request first, then send it to a prompt built for that kind of request. Support questions go one way, refund requests go another. Works when the categories are genuinely distinct.
\nParallelization. Running things at the same time, in one of two ways that constantly get mixed up. Sectioning splits a job into separate pieces that run side by side, which buys speed. Voting runs the same job several times and compares the answers, which buys confidence.
\nOrchestrator-workers. One model reads the job, breaks it into pieces on the spot, hands each piece to a helper model, then puts the answers back together. This is the one people mean when they say subagents.
\nEvaluator-optimizer. One model writes, a second marks it, and it loops until the work is good enough. Works when you can say clearly what good means.
\nReal systems mix these. A router at the front feeding into orchestrator-worker setups, each one checking its own output. The shape comes out of the job.
\nSubagents live inside the orchestrator-worker pattern, and the reason they exist is room, not horsepower.
\nA model can only hold so much text in mind at once. That limit is called the context window. Point one model at a research question with a hundred sources and it has to squeeze everything down to fit, and squeezing too hard loses the details that mattered.
\nSubagents get around that. Each one has its own window, works on a different piece at the same time, and sends back a short summary. The model in charge never has to hold all hundred sources at once.
\nThere's a second benefit that gets less attention. One agent that takes a wrong turn early tends to stay on that road, because everything it does afterward is built on the first bad step. Five agents looking independently don't share that wrong turn.
\nPeople use the two interchangeably. They're different.
\nIn orchestrator-workers, the helpers get created on the spot. The model in charge decides what this particular job needs, spins them up, and they're gone when it's done. No names, no history.
\nIn a multi-agent system the agents are permanent, and each one has a job. A research agent always does research. A critic agent always evaluates. They exist whether or not there's work to do.
\nGrok Bot's named bots are the second kind. Anthropic's research system is closer to the first. It matters because permanent specialists build up their own memory and their own bad habits over time, and temporary helpers aren't around long enough to.
\nIn June 2025, Cognition, the team behind Devin, published a post called Don't Build Multi-Agents. Their case: splitting a job across several agents is fragile, because each one only sees part of the picture and they end up making decisions that contradict each other. Their fix is what people call context engineering, which just means being deliberate about what information the model has in front of it at each moment.
\nThe next day Anthropic published How we built our multi-agent research system, describing close to the setup Cognition said not to build and arguing it was necessary for hard research questions. On their own research evaluation, a lead Opus 4 agent with Sonnet 4 helpers beat a single Opus 4 by 90.2%.
\nIt got read as a fight, and it partly is. They also agree more than the coverage suggested.
\nAnthropic's post says multi-agent is a poor fit when every agent needs the same information, or when the pieces of the job depend on each other. It names coding as the example.
\nCognition builds a coding agent.
\nSo the disagreement is narrower than it looked. Both think coding wants one agent with everything in front of it. Both think open-ended research, where you can go down several roads at once, is a different shape. They disagreed about which jobs, published a day apart, and got read as a holy war.
\nWhat I take from it is one question. Can the pieces of your job be done without knowing about each other? If piece B needs to know what piece A found, splitting them costs more than it buys.
\nThe most careful work here is a study called MAST, from a Berkeley team led by Mert Cemri, with Matei Zaharia, Joseph Gonzalez and Ion Stoica among the co-authors. They recorded seven popular multi-agent systems doing coding, math and general tasks, on both GPT-4 and Claude models. Human experts went through 150 of those recordings and built a catalog of the ways things went wrong, agreeing with each other almost every time, then applied it to more than 1,600 runs.
\nTheir opening line is blunt: despite the enthusiasm, the gains over single-agent systems on popular benchmarks are often minimal.
\nAcross those seven systems, between 41% and 86.7% of runs failed.
\nThey found 14 distinct ways to fail, in three groups. Roughly 42% came from the system being set up badly in the first place. About 37% came from agents talking past each other. About 21% came from nobody checking the work.
\nThe biggest group is setup, not capability. The two most common individual failures are agents redoing work they had already done, and an agent reasoning its way to the right answer and then doing something else.
\nA 2026 paper by Dat Tran and Douwe Kiela ran one agent against several on questions that take a few steps to answer, and did the thing most comparisons skip. It gave both sides the same amount of thinking to spend. Across three families of models, the single agent matched or beat the multi-agent versions almost every time. Which suggests some multi-agent wins are really the multi-agent version being allowed to think for longer.
\nGoogle Research tested five different setups across three model families and followed what happens to one small mistake. Agents working in parallel with nobody checking them amplified errors 17.2 times over. Put one agent in the middle whose job is to check everything before it moves on, and that drops to 4.4 times. If you split work, something in the middle has to check.
\nThen the cost. Anthropic's own numbers: a multi-agent setup uses about 15 times the tokens of a single chat, and a plain agent about 4 times. That's a budget decision.
\nThe harness is everything wrapped around the model. Which tools it can reach, how its instructions get put together, how it actually runs things, and what it sees when something fails. Same model, different harness, different behavior.
\nRecent evaluation work found the harness explains more of the difference in results than the choice of model does, and that the same model scores differently depending on which one it's sitting in. The recommendation is to report a score as model-plus-harness rather than crediting the model on its own.
\nSo when someone tells you a model is good at agent work, ask which harness. The claim doesn't mean much without it.
\nAgentic system. Umbrella term. Means almost nothing on its own.
\nWorkflow. You wrote the steps. The model does the thinking inside them.
\nAgent. The model decides the next step while it's running.
\nAugmented LLM. A model that can look things up, use tools and remember. The building block.
\nContext window. How much text a model can hold in mind at once.
\nSubagent. A temporary helper created for one job, with its own context window, gone when the job is done.
\nOrchestrator-worker. The setup where that happens. One model in charge, several helpers.
\nMulti-agent system. Permanent agents with names and specialties.
\nHarness. Everything around the model. Tools, instructions, how it runs, what it sees when it fails.
\nTool use. The model calling a function you gave it. Not autonomy on its own.
\nContext engineering. Being deliberate about what information the model has in front of it at each step.
\nReAct. The loop most agents run on. Think, do, look at what happened, repeat.
\nStart with the simplest thing that passes your tests. That's Anthropic's advice, and the MAST numbers back it up, because most failures came from bad setup rather than a weak model.
\nAsk whether the pieces of your job are genuinely independent. If piece B needs what piece A found, you want one agent with its information well managed, not five agents and a coordination problem.
\nIf the pieces are independent and the work is broad and open-ended, subagents earn their cost. Research is the clearest example. Coding mostly isn't, and both sides of the famous argument agree on that.
\nIf you do split the work, put something in the middle to check it. The gap between errors growing 17 times over and errors growing 4 times over is a checking step.
\nAnd budget for roughly 15 times the tokens before deciding multi-agent is worth it.
\nThe numbers say most of these systems fail because of how they were set up, not because the model wasn't smart enough. So the fix for a broken agent system is usually clearer instructions and better boundaries between tasks, and almost never a bigger model.
\nAnthropic, Building Effective Agents (December 19, 2024) for the workflow-versus-agent distinction, the augmented LLM and the five patterns. Anthropic, How we built our multi-agent research system (June 13, 2025) for the subagent rationale, the token multiples, the 90.2% eval result and the coding caveat.
\nCognition, Don't Build Multi-Agents by Walden Yan (June 2025).
\nCemri et al., Why Do Multi-Agent LLM Systems Fail? (arXiv 2503.13657, NeurIPS 2025) for MAST, the failure categories and the failure rates. Tran and Kiela, Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets (arXiv 2604.02460). Kim et al., Towards a Science of Scaling Agent Systems (Google Research, arXiv 2512.08296) for the error amplification figures. Harness findings from Harness-Bench (arXiv 2605.27922).
\nAnything wrong here is mine.
", "image": "https://quionie.com/og/ai-agent-architectures.png", "date_published": "2026-08-22T12:00:00.000Z", "date_modified": "2026-08-22T12:00:00.000Z", "tags": [ "AI agents", "agent architecture", "multi-agent systems", "subagents", "orchestrator-workers", "agentic workflows", "context engineering", "agent harness" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/grok-bot-vs-claude-cowork-vs-chatgpt", "url": "https://quionie.com/blog/grok-bot-vs-claude-cowork-vs-chatgpt", "title": "Grok Bot vs Claude Cowork vs ChatGPT", "summary": "Grok Bot, Claude Cowork, ChatGPT, OpenClaw and Hermes compared on where each one runs and whose credentials it holds, with every claim checked against the primary docs.", "content_html": "Grok Bot launched ten days ago and I've now read about thirty comparison articles on it. Most are written by companies selling a competing agent. One reviewer counted: nine of the fifteen top-ranking "best AI agents" guides are published by agent vendors, and all nine rank their own product first.
\nSo I went through the primary docs on all five instead. Grok Bot, Claude Cowork, ChatGPT, OpenClaw, Hermes. On one of them the summaries going around don't match the documentation.
\nI'm not selling anything, and I've dated the claims that move.
\nCowork went out in January as a desktop research preview, then to web and mobile on July 7. OpenClaw went viral in late January. Hermes shipped February 25. ChatGPT's agent launched July 9 on GPT-5.6. Grok Bot is the newest, on August 11.
\nThe positioning lines up almost exactly. A teammate, not a chatbot. You give it a goal instead of a prompt. It keeps working after you close your laptop.
\nFive labs with different incentives landed on the same shape inside eight months. That says more about what the technology currently allows than about any one product team. So the pitch won't separate them. Where each one runs will.
\nEvery one of these has to solve the same problem, which is that an agent needs to touch your stuff. There are three ways to do that and each product picks one.
\nRun on a cloud machine and log in as you. The agent gets a browser and uses your real accounts through the real web interfaces. Reaches everything, including software with no API. Holds your actual credentials. That's Grok Bot.
\nRun on your machine, on files you hand it. Local sandbox, reaching outside through connectors you approve. Less reach, much smaller blast radius. Cowork does this, and so do OpenClaw and Hermes, except self-hosted with no vendor in between.
\nRun in the vendor's cloud through official connectors. No credential handoff, no browser puppeteering, only the integrations that already exist. Closest to ChatGPT.
\nOnce you know which bucket something is in, you can predict its price, its reach and its failure modes.
\nIt's from SpaceXAI now, the entity that came out of SpaceX acquiring xAI in February. In June, SpaceX agreed to buy Anysphere, the company behind Cursor, for $60 billion in stock. That deal closed in the middle of this month, days before I wrote this.
\nThat matters, because the product ships through Cursor's stack. Per xAI's own docs, authentication, privacy settings, retention and deletion all resolve to Cursor rather than xAI.
\nThe product itself is good. Named persistent bots, each with a system prompt, on a cloud Linux machine with browser, filesystem and terminal. Two features nobody else shipped this cleanly. It will record up to ten minutes of you doing a workflow, browser only and no audio, and turn it into a draft skill you review. And bots message each other asynchronously, with group chats of two to six and the handoff visible in the transcript.
\nThe short write-ups mostly get the next part wrong.
\nAll of an account's bots share one persistent cloud computer. They share its files, its browser sessions and its app logins. Each bot gets its own screen on that one machine, which is a work surface rather than an isolation boundary.
\nStraight from the docs: "Do not use separate Bots as a security boundary." That sentence appears word for word on both the Grok Bot FAQ and the approvals and security page. xAI documented the constraint plainly, which is more than most of the coverage managed.
\nThe practical version: sign into a bank, a payroll system or anything holding regulated data on that machine, and every bot on your account can reach that session. If you pictured one bot per machine, correct that before you connect anything.
\nThe model powering Grok Bot isn't publicly named anywhere, and press guesses at a version number. For a product you hand credentials to, I'd want that stated.
\nOn price, be careful with the numbers going around. Access is still the same three tiers it was on day one: SuperGrok Heavy at $300 a month, Cursor Ultra at $200, and Cursor Teams Premium at $120 a seat. SuperGrok Plus and Cursor Pro+ exist and do not include Grok Bot. No free tier, and the seven-day trial some posts mention doesn't appear in the documentation.
\nSeven months of hardening versus Grok Bot's ten days, which is not nothing in a category this new.
\nArchitecturally it's the mirror. Cowork runs in a containerized Linux sandbox on your own machine, works on files you explicitly grant, and reaches services through connectors and a Chrome extension. Sandbox traffic is forced through a proxy with an allow list, and connector tokens never enter the sandbox. Grok Bot runs on a cloud computer it owns and gets to your tools through the same web interfaces you would use, no integration required.
\nGrok Bot reaches more, because a browser can log into anything. Cowork reaches less and knows exactly what it touched.
\nPrice gap is big too. Cowork rides along with Pro, Max, Team and Enterprise, so $20 is the door. Grok Bot's cheapest entry is six times that.
\nWhere Cowork can't help: if your workflow lives in a system with no connector and no API, it can't get there. That's the hole Grok Bot was built for. One old complaint about it is now out of date, though. Since the July expansion, a task started on desktop keeps running in the cloud with the laptop shut.
\nThe least flashy of the five, and probably the one most enterprises land on.
\nOpenAI ships the agent side of ChatGPT as ChatGPT Work, launched July 9 on GPT-5.6. Works over connected apps and files, produces documents, spreadsheets, presentations and web apps, and keeps running in the cloud after you close the app. OpenAI frames it as moving ChatGPT past answering questions and toward finishing work, which is close to word for word what SpaceXAI says about Grok Bot.
\nIt does drive a browser, but not a logged-in one. The cloud browser handles public pages; anything authenticated goes through connectors. That's the whole architectural difference from Grok Bot.
\nBetter pick when your tools have solid connectors and you mostly need messy inputs turned into something reviewable. Safer for anything stakeholder-facing.
\nIt runs subagents too: the parent spawns specialists in parallel, routes the work and merges the results. That makes multi-agent less of a Grok Bot differentiator than it looked on launch day.
\nOpenClaw is open source and self-hosted, built by Peter Steinberger in November 2025 as Clawdbot. It was renamed twice in four days under trademark pressure from Anthropic, first to Moltbot on January 27 and then to its current name on January 30. It went viral immediately, gaining around 25,000 GitHub stars in a single day and passing React's ten-year record in about sixty. Steinberger joined OpenAI on February 15 to work on personal agents, and the project moved to an independent foundation sponsored by OpenAI, NVIDIA, Microsoft and Tencent. As he put it himself, OpenAI hired him, not OpenClaw.
\nThen it became the first real AI agent security crisis. Every product in this post has the same shape of exposure. OpenClaw is the one with a public record.
\nCVE-2026-25253 was a one-click remote code execution via cross-site WebSocket hijacking: a crafted URL parameter made the control panel connect to an attacker's server and hand over its auth token. CVSS 8.8, patched at the end of January. It wasn't isolated. Command injection, server-side request forgery, path traversal that let attackers read local files, and prompt-injection-driven code execution all landed as separate CVEs. Later came a privilege escalation at 9.9 and a sandbox-escape race condition at 9.6.
\nThe plugin registry got hit too. Koi Security audited all 2,857 skills on the registry and found 341 malicious ones, 335 of them part of a single campaign pushing a macOS infostealer. Over a hundred posed as crypto wallet tools. Later audits at larger registry sizes kept finding malicious packages in the hundreds, though the rates reported vary enough that I wouldn't quote a single one.
\nExposure counts are all over the place. Censys counted 63,070 live instances at the end of March. SecurityScorecard found 42,900 exposed control panels across 82 countries. Earlier scans in January found a few hundred, later ones ran into six figures. I'm not picking a number, because they don't agree with each other. What is consistent across all of them is that most exposed instances had no authentication at all, because early versions bound to every network interface with no login required.
\nMicrosoft's security team wrote in February that OpenClaw "should be treated as untrusted code execution with persistent credentials" and "is not appropriate to run on a standard personal or enterprise workstation." Kaspersky said use a spare machine or a VPS, never your primary. NVIDIA shipped an enterprise security wrapper for it at GTC in March. China's industry ministry issued a public warning that bad configuration could expose users to attacks and data leaks.
\nA normal application has a defined scope, so you can put controls at each boundary. An autonomous agent has no fixed scope by definition, because the whole point is that it can do anything you could do. That's true of the category, not of OpenClaw specifically. It got hit first because it was open, popular, and deployed by a lot of people faster than the defaults could keep up. To the project's credit the fixes shipped quickly and the defaults changed.
\nReleased February 25 by Nous Research, MIT licensed, self-hosted, and widely treated as OpenClaw's technical successor. It ships a migration command that imports OpenClaw personality files, memories, skills and API keys straight over.
\nThe real difference is how much a human stays in the loop. Both learn skills from work you've already done. OpenClaw holds a learned skill as pending until you review and apply it, while Hermes writes and starts using them on its own after a handful of tool-heavy tasks. Memory lives in a local SQLite database on your machine and skills sit alongside it as plain markdown files. Nothing goes through a third party.
\nAdoption is real. NVIDIA's own blog reported in May that it crossed 140,000 GitHub stars in under three months and was the most used agent in the world by OpenRouter's numbers. It's at roughly 234,000 today, up from about 47,000 in April, so treat any single figure as a snapshot with a short shelf life.
\nThen the timing. Nous shipped Bot Mode on August 18, three days ago: named reusable bots with their own model and memory that message each other. That's Grok Bot's flagship feature, in an open-source product, seven days after Grok Bot launched.
\nSelf-hosted means everything in the OpenClaw section is now your problem.
\nAgentic benchmarks exist and the labs publish on them. OSWorld and GAIA for computer use, SWE-bench and Terminal-Bench for coding. Anthropic, OpenAI and xAI all report numbers.
\nThose numbers are for the models: Claude Opus, GPT-5.6, Grok 4.6. Nobody has published one for Grok Bot or ChatGPT or Cowork as shipped products, meaning the model plus its scaffolding, its browser, its permission prompts, and what it does when a page won't load. That scaffolding is most of what you're paying for.
\nSo when a review says one of them finishes the work, that's someone's impression. It might be right. It isn't a measurement, and that includes every review I read for this post.
\nWrite down what the agent has to touch. If everything on that list has an official connector, stop, and use the cheapest connector-based option. Most people are here and don't realize it.
\nIf your work lives in something with no API, which describes a lot of real work in construction, healthcare, logistics and government, browser-driving is the only thing that reaches it, and Grok Bot is the most polished version right now. You pay for that with real credentials on a machine you don't control, shared across your bots.
\nIf data can't leave your environment, you're self-hosting, and Hermes is the better-designed option with the more active project.
\nOn compliance, I expected this to be the tiebreaker and it isn't. All three of Anthropic, OpenAI and xAI hold SOC 2 Type II, and all three gate HIPAA agreements, SSO and zero-retention behind enterprise contracts and a sales conversation. xAI shipped its business and enterprise tiers in December. If you're in a regulated industry you're having the same procurement call three times, not picking a winner off a page.
\nNever pilot on the real account. Run it on low-stakes work and keep sensitive accounts disconnected until it proves out.
\nAnd price the failure, not the subscription. A $20 agent reading files you handed it and a $300 agent holding your CRM password carry very different downside.
\nWhether anyone benchmarks the products rather than the models, because until someone does, this market runs on assertion.
\nWhether Grok Bot names its model.
\nWhether the multi-agent thing means anything, given ChatGPT already had subagents and an open-source project matched the rest in a week.
\nAnd whether the first serious commercial-agent breach lands. OpenClaw took the hit because it was open, popular and deployed at speed. I don't think the polished ones are structurally safer. Fewer people have tried.
\nGrok Bot: xAI's product page, launch post and docs, including the FAQ on the shared computer and the security-boundary warning. Reporting on the SpaceX acquisition of xAI and the $60 billion Anysphere deal from CNBC, Forbes and TechCrunch.
\nClaude Cowork: TechCrunch on the July expansion and Anthropic's engineering writeup on sandbox containment. ChatGPT: InfoWorld on the launch, plus OpenAI's docs on subagents.
\nOpenClaw: Microsoft Security (February 19), Kaspersky, The Register on NVIDIA's wrapper, Reuters on China's ministry warning, Koi Security on the malicious skills, SonicWall and ARMO on the CVEs, Censys and SecurityScorecard on exposure counts.
\nHermes: Nous Research's repo and docs, and NVIDIA's blog (May 13) on adoption.
\nDated because most of these numbers moved in the last month, and a few moved while I was writing. Anything wrong here is mine.
", "image": "https://quionie.com/og/grok-bot-vs-claude-cowork-vs-chatgpt.png", "date_published": "2026-08-21T12:00:00.000Z", "date_modified": "2026-08-22T12:00:00.000Z", "tags": [ "AI agents", "Grok Bot", "Claude Cowork", "ChatGPT", "OpenClaw", "Hermes", "agent security", "AI tool comparison" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/how-large-language-models-work", "url": "https://quionie.com/blog/how-large-language-models-work", "title": "How Large Language Models Actually Work", "summary": "A large language model does one thing: predict the next token. This is the full mechanism, from tokenizers and parameters to why models hallucinate.", "content_html": "A large language model does one thing. Given some text, it produces a probability for what comes next.
\nI kept waiting to find out there was more to it. There isn't. Everything else, the reasoning, the code, the personality, the refusals, the essay it wrote you, comes out of that one operation run over and over.
\nMost of the confusion about these systems comes from skipping past that too fast. Slow down on it and a lot of arguments sort themselves out, including a few that serious people are still having in public.
\nFeed a model some text. It produces a probability distribution over its entire vocabulary for what token comes next. Not a word and not an answer, just a ranked list of every option with a number attached to each one.
\nThen something outside the model picks one, sticks it on the end, and feeds the whole thing back in. Over and over, until it produces a token that means stop.
\nThat loop is the entire operation. When a model writes you four paragraphs, it didn't plan four paragraphs. It picked a token, reread everything including what it just picked, and picked another one, a few thousand times.
\nWhich is strange, because the output holds together and the mechanism sounds like it shouldn't be able to do that. There's no outline, no goal state, no draft getting revised. Just a very good next-token guess, run enough times to add up to an essay.
\nBefore any of that, your text gets chopped into tokens, and tokens aren't words.
\nA tokenizer is a separate thing, built before the model and then frozen. It breaks text into chunks based on what was statistically common in the data it was built from. Common words are usually one token. Uncommon words get split. Spaces and punctuation get folded in.
\nSo the model never sees "strawberry." In OpenAI's tokenizer it gets three chunks, st and raw and berry, with no access to the letters inside them.
\nThat's the main reason models were bad at counting letters. Not because they can't count, and not because they're bad at math. You asked about a level of detail that got thrown away before the model saw anything. It's like asking someone to count the letters in a word they've only ever heard out loud.
\nIt isn't the whole story. Ask a model to spell the word out first and it usually can, because spellings appear in training text, and once the letters are separate tokens the counting works. So the failure is tokenization plus not thinking to break the word apart first.
\nNumbers get chopped the same way. OpenAI's tokenizer splits runs of digits into groups of up to three, reading left to right, so 1000 arrives as 100 and 0. A model doing arithmetic is working with units that don't line up with place value.
\nNot a bug in the model: A 2023 study covering about 200 languages found the same sentence can take 17 tokens in English and 198 in Burmese. You're billed per token and your context fills per token, so those users pay more and run out of room sooner for identical text. Nobody set that as a policy. It fell out of a statistical process nobody was watching.
\nWhen you read that a model has 70 billion parameters, a parameter is a weight, and a weight is a number saying how much one thing counts toward another. I wrote a whole study about that one idea, because the history of the field runs through it.
\nThat's the whole content of the model. A file of numbers.
\nParameters are what got adjusted during training and what stay fixed afterward. They aren't a database of facts and they aren't compressed text, though people reach for both metaphors. They're the settings that determine how input flows through to a probability distribution.
\nSo there's no lookup. When a model tells you a date, it isn't retrieving a stored date. The date falls out of the same arithmetic that produces everything else, which is why it can be wrong in ways a database never is.
\nAnd nobody picked any of the numbers. Not one. They're the residue of a training process no human supervised step by step, which is why nobody can open a model and read what it knows.
\nPretraining is the expensive part, and the recipe is boring.
\nTake enormous amounts of text. Hide the next token. Let the model guess. Measure how wrong it was. Nudge every parameter slightly in the direction that would have made the guess less wrong. Repeat over trillions of tokens.
\nThat's it. Nobody labels the data with what's true. Nobody teaches it grammar or facts or reasoning. The only signal is whether the next-token guess got better.
\nEverything the model appears to know is a side effect. To predict text well you have to encode a lot about how the world works, because text is about the world. Predicting the end of a mystery novel takes something like tracking who knew what. Predicting the next line of a proof takes something like following the proof.
\nWhether that "something like" adds up to understanding is the argument I end on. What isn't in dispute is the objective. It's next-token prediction, and it's the only thing being optimized.
\nFor a few years the assumption was that more parameters meant a better model. GPT-3 had 175 billion and that number was the headline.
\nIn 2022 a DeepMind team led by Jordan Hoffmann tested it properly. They trained over 400 models, from 70 million to 16 billion parameters, on 5 to 500 billion tokens. Their finding was that current large models were significantly undertrained. Too many parameters, nowhere near enough data.
\nThe rule they landed on: for a fixed compute budget, model size and training tokens should scale together, roughly 20 tokens per parameter.
\nGPT-3 was trained on about 300 billion tokens. That's under 2 tokens per parameter. It should have had roughly ten times more data, or been about ten times smaller.
\nThey proved it by building Chinchilla, 70 billion parameters on 1.4 trillion tokens, using about the same compute as a 280 billion parameter model called Gopher. Chinchilla beat Gopher, GPT-3, and Megatron-Turing NLG at 530 billion parameters. On MMLU it scored about 67.5 percent against Gopher's 60.
\nThe industry's central assumption was wrong for years, and the correction came from running the experiment properly rather than from a new idea.
\nThe number has moved again since. Llama 3's 8 billion parameter model was trained on 15 trillion tokens, which is about 1,875 tokens per parameter, nearly a hundred times the Chinchilla ratio. Meta did that on purpose and said why: compute-optimal minimizes the cost of training, and ignores that you then serve the model billions of times. Training is a one-time bill. Inference is forever. A smaller model trained far longer is cheaper for good.
\nThe correction got corrected. That's how this goes.
\nA freshly pretrained model, a base model, is not an assistant. It's a text continuation engine. Ask it a question and it might answer, or it might generate three more questions, because in the training data questions are often followed by more questions.
\nIt has no concept of being asked. There's no user, no turn, no task. It continues text, and a question is just text.
\nSo the thing OpenAI trained in 2020 wasn't something you could hand to the public. It knew an enormous amount and wouldn't reliably do anything with it.
\nThe fix is a second stage, and it costs almost nothing next to the first.
\nOpenAI's InstructGPT work in 2022 laid out the recipe everything since is a variation of. Collect about 13,000 prompts and have humans write good responses, then fine-tune on those. Have humans rank model outputs against each other, about 33,000 prompts worth, and train a separate reward model to predict those rankings. Then tune the model to score well against that reward model.
\nHuman raters preferred outputs from a 1.3 billion parameter InstructGPT model over GPT-3 at 175 billion, a model with a hundred times more parameters. At matched size, raters picked InstructGPT over GPT-3 about 85 percent of the time. On closed-domain tasks, where the answer isn't supposed to contain anything that wasn't in the input, hallucination dropped from around 41 percent to around 21.
\nThen the compute. Pretraining GPT-3 took roughly 3,640 petaflop/s-days. The reinforcement learning stage took about 60. Under two percent of the pretraining budget did more for usefulness than a hundredfold increase in size would have. The paper says so directly: RLHF is more effective at making models helpful than a 100x model size increase.
\nThat's the stage where the assistant gets made. It's also where most of what people complain about gets installed, because you're optimizing the model to produce what human raters approved of.
\nSycophancy comes at least partly from here. Anthropic found that both human raters and the preference models trained on them tend to prefer responses matching the user's stated view, sometimes over correct ones. The flattened writing voice, the hedging, the particular shape of how a model says no, same stage. None of that is emergent. It's a preference signal, applied on purpose, by people making judgment calls.
\nIt's also a large part of why models from different labs feel different, which is worth its own section.
\nThey're the same kind of thing. Every frontier chat model is a transformer predicting the next token, pretrained on an enormous pile of text and then shaped by a preference stage. No lab has a fundamentally different recipe.
\nWhat differs is everything downstream of that.
\nThe training data. Undisclosed almost everywhere. Meta is the exception and only roughly, publishing Llama 3's mix as about half general knowledge, a quarter math and reasoning, 17 percent code, 8 percent multilingual. For OpenAI, Anthropic, Google and xAI you get nothing usable. It's a real difference nobody outside the lab can measure.
\nThe preference stage. This is where they actually diverge, and it's the best documented part. Anthropic uses Constitutional AI, where the model critiques its own output against a written set of principles rather than relying only on human raters, plus a separate character training stage aimed at the persona. OpenAI publishes a Model Spec, a document telling raters what good behavior looks like, which then becomes the preference data. Different instructions, different model, similar starting material.
\nThe system prompt. A block of text stuck in front of your conversation that sets tone and rules. It isn't in the weights and it can change between Tuesday and Wednesday. Anthropic is the only major lab that publishes theirs with a changelog. What circulates for the others is leaks of uncertain accuracy.
\nThe tokenizer. Different vocabularies chop your text differently. GPT-4o's has about 200,000 entries, Llama 3's has 128,256, Google's Gemma family around 256,000. Anthropic has never published Claude's. Same sentence, different token count, different bill.
\nThen there's engineering that changes cost and speed without changing the kind of thing. Some models are mixture-of-experts, routing each token through a fraction of the network instead of all of it. DeepSeek-V3 has 671 billion parameters and uses 37 billion per token. Llama 4 and Gemini do versions of this. OpenAI and Anthropic don't say what they do.
\nAnd a lot of what feels like a difference isn't the model. Memory, web search, code execution, file handling, artifacts. That's the product wrapper, which gets its own section further down.
\nWhat nobody can tell you is which of these produces which personality. There's research on how to steer a model's personality once you have one, and nothing rigorous comparing the shipped products to explain why they land differently on people. So when someone tells you Claude is thoughtful and GPT is creative, that's a real impression with no established cause behind it.
\nHallucination gets discussed as a bug better engineering will eventually remove. The research is less comfortable than that.
\nA 2024 paper by Adam Kalai and Santosh Vempala, Calibrated Language Models Must Hallucinate, proved a floor. For arbitrary facts, meaning ones you can't work out from patterns in the data, a calibrated model has to produce falsehoods at some minimum rate. The bound comes from calibration itself, not from bad data or the transformer architecture. It's roughly the fraction of facts that appeared exactly once in training.
\nWorth being precise about the scope. That floor applies to a calibrated pretrained model, and the paper points at post-training as the way underneath it, since post-training deliberately gives up strict calibration. So it isn't a permanent ceiling on shipped products. It's a property of the pretraining objective.
\nTheir 2025 follow-up with OpenAI went after why post-training doesn't finish the job, and found the rest of the problem in incentives. Generating a right answer is harder than recognizing one, and they formalized the gap: the generation error rate is at least twice the classification error rate, minus a calibration term. If a model can't reliably tell true from false for some fact, it certainly can't reliably generate the true one.
\nThen the part that's a human problem rather than a math problem. They surveyed the major leaderboards and found most of them score answers as simply right or wrong, which means "I don't know" scores exactly the same as a confident lie. A model that guesses beats a model that abstains, every time, on every board. We trained these things to be test-takers and then acted surprised when they bluff.
\nTheir fix isn't a new architecture. It's changing the scoring. Put an explicit confidence threshold in the question, answer only if you're more than that confident, and penalize a wrong answer in proportion to the threshold you were given. One of the field's most notorious failures turns out to be partly a measurement culture problem, and the people who study it are the ones saying so.
\nYou've probably heard that models gain abilities suddenly at certain sizes. A capability is absent, absent, absent, then a model crosses a threshold and it's there. Wei and colleagues made the case in 2022 and called them emergent abilities, with a companion catalog listing 137 examples. It became one of the most repeated claims about LLMs, and it carries real weight, because unpredictable capability jumps are a safety argument.
\nIn 2023, Schaeffer, Miranda and Koyejo argued the jumps are largely an artifact of measurement. Score with something all-or-nothing, like exact string match on an arithmetic problem, and smooth underlying improvement looks like a sudden leap. Switch to a metric with partial credit and the same data draws a smooth curve. Over 92 percent of the emergent abilities annotated on BIG-Bench showed up under just two scoring rules, both of them all-or-nothing. The paper won an Outstanding Paper award at NeurIPS that year.
\nIt's a good argument and it hasn't ended the debate. A 2025 survey by Berti, Giorgi and Kasneci pushes back, pointing out that a log axis can manufacture the appearance of smoothness, and that some of the smoothed curves still contain jumps from under 10 percent to near 100 percent accuracy across a single step in scale. Their question is fair: does going from 10 percent to 100 percent stop being a jump because of how you drew the axis?
\nThis isn't settled, and I'd be careful of anyone who says it is in either direction. What's fair to say is that the confident version, where capabilities appear from nowhere at unpredictable scales, is weaker than it was in 2022, and how much weaker depends on measurement choices people are still arguing about.
\nThe model does not remember your conversation. Every turn, the entire transcript gets fed back in and processed from scratch. Nothing carries over inside the model, because the weights are frozen and reading text doesn't change them.
\nBut products built on models do have memory, it works, and people use it constantly. When you tell an assistant to remember something, that gets written to a database outside the model as text. Next time, it gets retrieved and pasted into the input alongside your message.
\nBoth things are true, and the distinction is the useful part. The remembering happens outside the intelligence. It's a retrieval system deciding what to paste in, which is why memory features behave the way they do. You can read them, edit them, delete them, and they sometimes surface the wrong thing or miss the obvious thing. Those aren't glitches in a mind. They're retrieval decisions.
\nSame logic for the rest of the wrapper. When the thing searches the web, the model didn't do that. When it runs code, executes a task, reads your file, the product did that and handed the result to the model as more text.
\nPeople attribute product features to the model and then draw conclusions about the model. Most of what looks like agency is plumbing.
\nA large language model is a very large set of numbers, adjusted by a process that repeatedly guessed the next chunk of text and got corrected, until those numbers encoded enough about language and the world to make the guesses good. Then a second, much cheaper process shaped it into something that answers instructions the way human raters preferred.
\nYou interact with it by feeding text in and taking tokens out, one at a time, with a sampler choosing among ranked options. Around it sits software providing memory, tools, search and safety filtering, none of which the model itself has.
\nThat's the honest description. No step in it requires the model to understand anything, want anything, or know that you exist.
\nThree corrections, and the last one runs against the other two.
\nIt isn't a database. Facts aren't stored and retrieved. They fall out of the same arithmetic as everything else, which is why a model can be confidently wrong about a date in a way no lookup system ever is, and why "just make it only say true things" isn't a coherent engineering request.
\nIt isn't a person. Nothing persists between conversations. No continuity, no accumulation, no self that grows. The apparent personality is largely the output of a preference-tuning stage, and it can be changed by retraining without anything in there noticing.
\nAnd it isn't obviously just autocomplete either, which is where I part ways with the confident dismissal. The autocomplete description is mechanically correct, and it smuggles in an assumption: that a system trained to predict text can only ever be shuffling text around. That's an empirical question, not a definitional one, and the honest answer is that we don't know what the training produced, because nobody can read the numbers.
\nThe dismissive version and the mystical version make the same mistake. Both claim to know what's inside from the outside.
\nWe know exactly how it works and we still don't know what it is. That's an uncomfortable place to leave it, and it's where things actually are.
\nHoffmann et al. (2022), Training Compute-Optimal Large Language Models, for Chinchilla and the 20-tokens-per-parameter result; Meta's Llama 3 post and Sardana et al. (2024), Beyond Chinchilla-Optimal, for training past it on purpose. Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback, for InstructGPT and the compute figures. Sharma et al. (2023), Towards Understanding Sycophancy in Language Models. Kalai and Vempala (2024), Calibrated Language Models Must Hallucinate, and Kalai, Nachum, Vempala and Zhang (2025), Why Language Models Hallucinate. Wei et al. (2022) on emergent abilities, Schaeffer, Miranda and Koyejo (2023), Are Emergent Abilities of Large Language Models a Mirage?, and Berti, Giorgi and Kasneci (2025) for the counterargument. Petrov et al. (2023), Language Model Tokenizers Introduce Unfairness Between Languages. Anything wrong here is mine, not theirs.
", "image": "https://quionie.com/og/how-large-language-models-work.png", "date_published": "2026-08-14T12:00:00.000Z", "date_modified": "2026-08-22T12:00:00.000Z", "tags": [ "large language models", "transformers", "tokenization", "neural networks", "AI hallucination", "how AI works" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/mental-models-worth-keeping", "url": "https://quionie.com/blog/mental-models-worth-keeping", "title": "The Mental Models I Keep Coming Back To", "summary": "The mental models I actually reach for, from first principles to compounding to leverage, what each one is good for, and the point where each one turns dangerous.", "content_html": "Most people collect advice. Almost nobody collects better ways of thinking.
\nThat's backwards. Advice is glued to the situation it came from, so it expires the second the situation changes. A way of thinking travels. You learn it in one place and it keeps working somewhere nobody expected.
\nThat's what a mental model is. A compressed version of how some part of the world behaves, portable enough to use somewhere it wasn't built for. Compound interest is a fact about money. It's also a fact about skills, reputation, and grudges. Same shape, different material.
\nThe problem is that mental models turned into a collectible. Somewhere in the last decade people started hoarding them like productivity apps. A hundred concepts, each one flattened into a tweet, none of them ever pointed at an actual decision.
\nThat's not what Charlie Munger meant. He wasn't handing out flashcards. He was describing a habit: reach for the few models that explain what's in front of you, and notice the second they stop.
\nSo this isn't a list of a hundred. It's the ones I actually use, including the parts where they turn on you. Every one of them breaks somewhere. The people who get burned usually learned it as a rule and never went looking for where it stops.
\nThe idea is old and simple. Reason up from things you know are true instead of sideways from what everyone else is doing. Aristotle called it the first basis from which a thing is known. In practice it means refusing to treat a number as fixed just because the market agreed on it.
\nBattery packs are the famous case. Take the going price as a law of nature and electric cars stay expensive forever. Price the raw materials instead, the nickel and cobalt and aluminum sitting on the commodity exchange, and the floor turns out to be way below what anyone was quoting.
\nThat's the good version. The bad version is a personality. Reasoning up from physics is a great story for a biography and a terrible way to pick a restaurant. Rebuilding a field from scratch is slow and expensive, and the convention you're sneering at is usually hard-won knowledge from people who already paid for the mistakes you're about to make. First principles becomes a party trick the moment the derivation gets more fun than the decision.
\nSo use it on one thing at a time. Find the single assumption everyone treats as permanent, and ask who decided it, when, and whether they'd decide the same today. That's usually where the mispriced thing is hiding. Everywhere else, take the shortcut. Convention is a decent default, and it's a lot cheaper than rederiving the world before lunch.
\nWhere it works: Point it at the one number a whole industry treats as fixed. Take the shortcut on everything else.
\nCarl Jacobi supposedly told his students to invert, always invert. Munger built his whole style on it. All he wanted to know was where he was going to die, so he could avoid the place. Instead of asking how to win, ask how you'd guarantee losing, then don't do those things.
\nIt works because failure is smaller and more concrete than success. Nobody can hand you a recipe for a great company or a good marriage. Everybody can tell you how to wreck one. Avoiding stupidity is more reliable than chasing brilliance, and you can start on it today.
\nThe cleanest version is the pre-mortem. Before you start, pretend it's a year from now and the whole thing has already failed, then write the story of how. People will say things in that exercise they'd never say in an optimistic kickoff, because you've made it safe to be the one who saw it coming instead of the one killing the mood.
\nThe limit is built in. Inversion tells you what to avoid, not what to build. Spend your entire life dodging mistakes and you end up with a very clean, very small life. It's a filter, and a filter needs something behind it.
\nThe pre-mortem: Pretend it's a year from now and the project already failed, then write down exactly why. People name risks there they'd never raise in an optimistic kickoff.
\nHoward Marks) has the sharpest version of this. First-level thinking says the company is good, buy the stock. Second-level thinking says everyone already knows it's good, so it's priced like it, and the only money left is in the gap between what people expect and what actually happens. Being right about the obvious pays nothing. The obvious is already in the price.
\nUnderneath it is a plainer idea. Consequences have consequences. Rent control drops rents for the people who already have an apartment, then kills the reason to build any more, so rents climb for everyone still stuck outside. The first effect is the one on the poster. The second and third are the ones that actually run the world, and almost nobody reads that far down.
\nThe failure mode is that you can always ask "and then what" one more time. Do it forever and you've talked yourself out of every decision and called it rigor. At some point the next loop costs more than it's worth and you just move. Second-order thinking done badly is anxiety with a flowchart.
\nThe catch: You can always ask "and then what" one more time. Past a point that's not rigor, it's paralysis.
\nKahneman and Tversky spent years proving that people ignore base rates in favor of a good story. Describe a shy, tidy, detail-obsessed man and ask whether he's more likely a librarian or a farmer. Most people say librarian and forget there are far more farmers in the world. The story beats the math.
\nThe fix is what forecasters call the outside view. Before you guess how long your project will take, ask how long projects like it usually take, and start there. Your version is a story you're the main character in. The base rate is the record of everyone who tried the same thing and also figured they were the exception.
\nTetlock's superforecasters basically do this on autopilot. Start at the base rate, update in small steps as real evidence shows up, and don't lurch at the first dramatic headline.
\nThe part nobody selling a productivity system will admit is that more information doesn't reliably make your decisions better. Past a surprisingly low point, extra detail grows your confidence faster than your accuracy. You feel more sure and you're no more right, which is worse than knowing nothing, because now you'll bet big on it.
\nBase rates only mislead when there's no real comparison group, and that's rarer than people want it to be. It usually gets claimed by someone who really needs their case to be special.
\nThe trap: More information mostly grows your confidence, not your accuracy. Start from what usually happens, then adjust.
\nExpected value is probability times payoff, added up over the outcomes. Obvious on paper, ignored in real life, because a loss stings more than the same-size win feels good, so people pass on bets they should take.
\nThat instinct has a name, loss aversion, and it comes with a caveat worth knowing. The tidy claim that a loss hurts exactly twice as much as a matching gain has taken a real beating in replication lately. It's more situational than the paperbacks make it sound. It's still there, and it still explains a lot of timid choices. It's just softer and stranger than the number everyone repeats.
\nThe more useful cousin is asymmetry. You don't have to be right often if you're right big and wrong small. When the worst case is small and fixed and the best case is huge and open-ended, a startup, a cold email, an essay you put on the internet, you can miss most of the time and still come out ahead, as long as you survive the misses.
\nBenjamin Graham's margin of safety is the same instinct made concrete. Build the bridge to hold thirty tons and only drive ten-ton trucks over it. The gap is there to absorb the error in an estimate that's wrong somewhere you can't see.
\nThen there's the rule that sits on top of all of it. You have to still be in the game. Expected value assumes you get to keep playing. A bet with great expected value and a five percent chance of ruin is a bad bet when you only get one life to run it in, because the average across a thousand imaginary versions of you means nothing to the actual one who hit zero. You can't compound from nothing. "Never risk what you can't afford to lose" isn't caution, it's math. Risk is only worth taking when the downside is survivable. When it's not, the upside doesn't matter, because you won't be there to collect it.
\nThe one rule: Positive expected value means nothing if a bad outcome takes you out of the game. You can't compound from zero.
\nEvery choice has a cost that never shows up on the receipt: the best thing you didn't pick. A dollar in a mediocre investment isn't only earning less, it's also not in the good one, and that second loss is invisible and usually bigger. Good investors judge every option against their next best one instead of against zero, which is why they pass on things that are merely fine.
\nTime works the same way and it hurts more. The real cost of a fine yes is the great thing you now can't say yes to, because you're busy with the fine one. For capable people the most expensive line item is the one that never appears anywhere. It's the work they'll never take, because they already said yes to something adequate.
\nThe trap is that this can curdle into never committing. Everything has an opportunity cost, so weigh it too hard and you keep every slot open and do nothing, which turns out to have the highest opportunity cost of all. The tool is for choosing better. It's not for making the act of choosing unbearable.
\nThe catch: Weigh it too heavily and you commit to nothing, which is the most expensive choice of all.
\n"Show me the incentive and I'll show you the outcome," Munger said, and he thought it was one of the strongest forces he'd ever watched work. A lot of what looks like stupidity or malice is just someone responding sensibly to an incentive you can't see from where you're standing.
\nThe sharp edge of this is Goodhart's Law, best phrased by the anthropologist Marilyn Strathern: when a measure becomes a target, it stops being a good measure. Teachers told to hit test scores teach the test. Salespeople at Wells Fargo told to open accounts opened around two million fake ones. Measure a team on lines of code and you get software with the structural integrity of wet cardboard. The number was a fine stand-in for the thing you cared about, right up until you started paying people for the number. Then they optimized the number and dropped the thing.
\nThe part people skip is that the cynical read is also wrong. Assume everyone's a pure self-interested optimizer and you'll build a culture that produces exactly that, because people tend to become whatever you treat them as. Plenty of people do the right thing when it costs them. And over-measuring has a way of killing the quiet motivation that was doing most of the real work in the first place. Incentives explain a lot. They don't explain everything, and running a company as if they do is a reliable way to build one nobody wants to work at.
\nGoodhart's Law: The moment a measure becomes a target, it stops measuring the thing you cared about.
\nG. K. Chesterton wrote it as a scene. There's a fence across a road for no reason you can see. The reformer says, I don't see the point of this, let's clear it away. The wiser answer is: if you don't see the point, I definitely won't let you touch it. Go away, work out what it's for, come back, and then maybe I'll let you take it down.
\nThe lesson is to leave alone what you don't understand yet. The weird approval step, the legacy code everyone's scared to touch, the rule that looks pointless. Someone put it there, maybe for a reason that's no longer obvious but is still holding something up. Systems collect scar tissue, and scar tissue is usually covering an old wound.
\nBut watch how fast this turns into an excuse to never change anything. "Someone must have had a reason" is not itself a reason. The person who built the fence might have been an idiot, or solving a problem that stopped existing in 1994. The discipline isn't leaving the fence up. It's being able to say what it's for before you decide. Once you can, and the reason's dead, take it down and don't make a ceremony of it.
\nThe discipline: Don't touch the fence until you can say what it's for. Once you can, and the reason's gone, take it down.
\nThis is the one that earns the word everyone overuses. It runs on one specific mechanism. Each period's gain becomes the next period's starting point, so the growth builds on the growth, and the line stays boring for a long time before it stops being boring.
\nIt's not just money. Skills compound, reputation compounds, relationships compound. So do the ugly ones, debt and resentment and a body you keep ignoring, which is why a small bad habit is so much worse than it looks on any single day.
\nThe line usually pinned on Munger is that the first rule of compounding is to not interrupt it unnecessarily. Most of the damage people do to their own curve is self-inflicted. They sell at the bottom, quit in year three, torch a decade of trust for one good quarter.
\nBut compounding only matters if the thing underneath is worth compounding. Ten years of practicing the wrong technique makes you excellent at the wrong technique. A business growing twenty percent a year while it loses money on every sale is just going bankrupt with better momentum. People say "it compounds" to justify grinding on things that don't add up to anything, a job with no skill transfer, an audience that will never buy a thing, a pile of contacts they'd never actually call. Patience on the wrong asset isn't a virtue, it's a slow leak. Check that the thing compounds before you congratulate yourself for sticking with it.
\nThe catch: It only pays if the thing underneath is worth compounding. Patience on the wrong asset is just a slow leak.
\nLeverage is borrowed force. Debt is the obvious kind, but code, capital, other people's time, and an audience are all leverage too. They multiply whatever you point them at. The part people forget is that they multiply mistakes exactly as well as they multiply good calls.
\nWarren Buffett likes to say smart people go broke three ways: liquor, ladies, and leverage. He admits the first two are only in there because he needed words that start with L. A brilliant call and a terrible one both look fine while the borrowed money is still working for you. The difference only shows up when the loan comes due, and by then only one of them lets you keep playing.
\nOptionality is the opposite instinct, and the antidote to it: keep your options open. Favor bets you can't lose much on but could win big from, the things that get stronger from chaos instead of breaking under it. That last part is Nassim Taleb's idea of antifragility, and his barbell is the shape of it: most of your resources somewhere boring and safe, a small slice somewhere wild, and nothing parked in the respectable-looking middle that hides how fragile it is until the worst possible moment. Jeff Bezos framed the everyday version as one-way and two-way doors. Most decisions are reversible, so make those fast and cheap, and save the slow, careful kind for the few doors that only open once.
\nOptionality has its own failure, and it's a quiet one. It turns into never choosing at all. Keeping every door open is how you end up walking through none of them. And options aren't free. What you pay is the depth and focus you give up by staying available. At some point the antifragile move is to shut the doors and commit, because compounding, the one from earlier, only pays off if you stay put long enough to let it.
\nThe catch: Leverage multiplies mistakes as fast as wins, and keeping every door open is how you walk through none.
\nIf you want to learn from people who got rich, study the machine and skip the worship. The interesting thing about Costco isn't anyone's net worth. It's that a company that caps its own markup and makes most of its real profit on membership fees has built something almost impossible to undercut, because there's no margin left for a competitor to attack.
\nThat durability is what people mean by a moat. Network effects, where each new user makes the thing more valuable to everyone else, so a payment network that's useless with one merchant is unbeatable with a few million. Switching costs, the enterprise software nobody will ever find the will to rip out. Scale, where being the biggest just makes you the cheapest. Brand, which at its best is a promise strong enough that people stop comparing prices and pay more for the same molecule under a name they trust. None of that comes from the marketing department. It's structural, and structure is what gets you through a bad year.
\nFounders almost always overrate the product and underrate distribution. Peter Thiel's uncomfortable point is that a better product with no way to reach people loses to a worse product that reaches everyone. "Build it and they will come" is survivorship bias talking. You only ever hear the stories where it happened to work.
\nThe engine under all of it is ownership. The math is plain: a wage is linear and taxed hard on the way in, while a claim on an appreciating asset compounds and is taxed lightly, if at all, on the way out. That's why almost every large fortune traces back to owning something, equity or a business or property, more than to a paycheck. None of which makes a salary a bad deal. It's steady, it's low-risk, and it's usually what pays for any ownership in the first place. The two just grow differently, and the useful thing is understanding why.
\nThe failure here is copying the surface. Costco's food court won't give you Costco's economics, and Apple's font won't give you Apple's brand. You have to build what's underneath, and that part never copies.
\nThe mechanism: A wage grows in a straight line; an appreciating asset compounds. That's the whole difference, and it's just math.
\nBuffett talks about a circle of competence, and people usually miss his actual point. You don't need to be an expert on everything. You need to know where the edge of your circle is. The size of it barely matters. Knowing the border is the whole game, because the expensive mistakes almost always happen just outside it, dressed up to look like they're just inside.
\nLook back over the rest and they rhyme. First principles, base rates, margin of safety, Chesterton's fence, the circle itself. Most of them are disciplined ways of admitting what you don't know. Inversion is humility about your odds of being brilliant. Margin of safety, humility about your estimates. Second-order thinking, humility about consequences. The models aren't clever. They're just honest in a structured way.
\nNone of them holds up alone. The person who only knows inversion becomes a professional pessimist who never builds anything. The one who only knows compounding never sells, never exits, never cuts a loss. They work as a set, checking each other, and the skill is knowing which one the moment is asking for. That part is judgment, and no model can hand you judgment. It's what's left after the models have narrowed things down.
\nNone of these are the whole picture. Each one makes a specific thing easy to see and leaves the rest out, which is fine as long as you know that going in. Keep the few that hold up when you actually use them, and stay ready to drop one when reality stops agreeing with it. It always does eventually.
\nIf you want the originals instead of my read on them: Munger's Poor Charlie's Almanack and his 1994 talk on worldly wisdom; Howard Marks in The Most Important Thing; Graham's The Intelligent Investor; Kahneman's Thinking, Fast and Slow and Tetlock's Superforecasting for the base-rate work; Chesterton's The Thing for the fence; Taleb's Antifragile); and Thiel's Zero to One for distribution and power laws. Everything above is my read on these, so anything that's off is on me, not them.
", "image": "https://quionie.com/og/mental-models-worth-keeping.png", "date_published": "2026-07-07T12:00:00.000Z", "date_modified": "2026-07-07T12:00:00.000Z", "tags": [ "mental models", "decision making", "second-order thinking", "expected value", "risk of ruin" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/history-of-ai-one-number", "url": "https://quionie.com/blog/history-of-ai-one-number", "title": "A History of AI, Told Through One Number", "summary": "The part of ChatGPT that seems to think is nothing but numbers nobody chose. A history of AI told through one idea: the weight.", "content_html": "The part of ChatGPT that actually seems to think is nothing but numbers.
\nThere's code that runs it, and a hidden prompt that nudges its tone. But the ability itself, the part that can write and reason and hold a conversation, was never written down by anyone. No rules, no logic anyone typed in. It lives in hundreds of billions of numbers sitting in a file.
\nNobody picked them. Not one, not ever, and no human went through them by hand. We built the process that finds the numbers and handed over the part where you'd normally understand what you made.
\nThat fact sits at the bottom of every argument about AI right now, and almost nobody explaining this stuff starts there. They start with neural networks, or training data, or transformers, all of which are made of something smaller that rarely gets named.
\n\nA weight is a number that says how much one thing matters to another.
That's it. Learn that one properly and the last seventy years turn into a single argument with one question in it: who picks the numbers. Humans, or the machine.
\nYou already do this. You just don't call it weighting. Meeting someone for the first time, you decide how much to trust them by leaning hard on a couple of signals (whether they actually listen, how they treat the waiter) and ignoring others completely (what car they drive). Nobody handed you that ranking. You built it over years of misjudging people and adjusting.
\nThat ranking is the whole idea of a weight: how much a given signal counts toward the answer. An artificial neuron is the same move in numbers. Each signal is multiplied by its weight (high, low, or zero), the results are added up, and if the total clears a line, the neuron fires.
\nStack billions and you have a language model. The behavior has no second ingredient. No rules, no logic anyone wrote for what it should say. Just numbers saying how much things matter to each other, found by training rather than typed in by hand.
\nWhich is what makes the question of who sets them the only one that's ever mattered. Seventy years, two collapses, one long feud, and the thing on your phone. All of it is that.
\nWarren McCulloch, a neurophysiologist, and Walter Pitts, a logician who taught himself the field as a teenager and was more or less homeless when the two of them met, published a paper describing brain cells as simple logical switches. Inputs, a threshold, an output. On or off.
\nThey meant it as biology. What it proved by accident is that you can build thought-shaped things out of arithmetic.
\nIt couldn't learn, though. A person set the weights by hand, and that person had to already know the answer. Which makes it a description of a brain, not a thing that behaves like one.
\nThat gap took fifteen years to close.
\nFrank Rosenblatt was a psychologist at Cornell, and the piece he added is almost insultingly simple.
\nShow it an example. Let it guess. If the guess is wrong, nudge the weights slightly toward what would have been right. Do it again.
\nThat's the perceptron learning rule, and it is the ancestor of everything running today. Nobody tells the machine what matters. It finds out by being wrong at scale.
\nHe first ran it as software on an IBM 704, and the Navy held a press conference in 1958 to show it off. The New York Times covered it under a headline about a device that learns by doing, describing a machine expected to eventually "walk, talk, see, write, reproduce itself and be conscious of its existence." The New Yorker called it the first serious rival to the human brain ever devised.
\nWhat the machine had actually done that day was learn to tell left from right, after about fifty tries.
\nThe hardware version came around 1960. The Mark I Perceptron was room-sized, with a 20x20 grid of photocells for an eye, wired to banks of motor-driven potentiometers that served as the adjustable weights. Which means the weights were knobs, and learning was a set of small motors turning them. You could stand in the room and watch a machine physically change its mind.
\nEvery hype cycle in AI since has the shape of that press conference. A real breakthrough, correctly spotted as a big deal, then described in language that runs ahead of what the thing can do yet. Worth holding onto when you read anything about AI now, including this.
\nTake a small logic puzzle called XOR. Two inputs, each either on or off. You want a yes when they differ (one on, one off) and a no when they match. It's the logic of a hallway light wired to a switch at both ends: flip either switch and the light flips. Sounds like nothing.
\nOne layer of weights cannot do it. Not "hasn't managed yet." Cannot, provably.
\nThe reason is geometric. A single layer can only draw a straight line through your data, one side yes and the other no. XOR needs a division a straight line can't produce. And once you know to look for that shape, you find it in most problems worth solving.
\nThe fix was visible even then. Stack layers and the network can bend the line. The catch was that nobody could train a stack. The learning rule works on one layer because when the answer comes out wrong you know precisely which weights to blame. Add layers and the blame goes murky. Which weight, in which layer, caused this?
\nSo the field had a machine that could learn but not handle anything real, and a design that could handle real things but not learn.
\nWhy layers matter: One layer of weights can only separate things with a straight line. Most real problems need a curve, and a curve takes stacked layers. The whole catch was that nobody could train the stack yet.
\nMarvin Minsky and Seymour Papert published a book called Perceptrons) laying out exactly that limit, and the standard telling is that it single-handedly killed neural networks and triggered the first AI winter. One book, funding gone, field dead.
\nThat version is in most explainers and it doesn't survive contact with the sources.
\nMinsky and Papert had been arguing against this approach since around 1965, at conferences and in circulated drafts. By 1969 most researchers had already drifted away, worn down by lack of progress, and the rules-based camp had largely won the funding fight before the book existed. It landed in a room that was already emptying and picked up its reputation as the assassin partly through timing.
\nRosenblatt died in a boating accident in Chesapeake Bay in July 1971, on his 43rd birthday. When the book was reissued in 1987, it carried a dedication to him.
\nWorth knowing which version you're repeating, because the tidy one makes it sound like ideas die from criticism. They mostly die from people quietly leaving.
\nLeft out of most tellings: for about three decades the mainstream of AI wasn't weights at all.
\nThe other approach was rules. Write down what you know as explicit logic and let the machine reason over it. If fever and cough, consider flu. Knowledge as statements a human typed in.
\nThis was the sensible bet, not the dumb one. Rules are readable. You can debug them. You can ask the system why it decided something and get a real answer, which is the exact thing everyone now complains AI can't do.
\nIt produced expert systems that worked in narrow slices (diagnosing infections, configuring hardware orders), and a genuine industry grew around them in the 80s.
\nThen it hit the wall it was always going to hit, which is that a human has to type in every single thing. Every rule, every exception, every exception to the exception. Systems grew past a few thousand rules and started contradicting themselves in ways nobody could untangle. And the parts we do without thinking, seeing, hearing, reading a room, turn out to be the hardest to write down, because nobody can say what rules they're following.
\nThe market collapsed in the late 80s. Second winter.
\nBoth approaches failed, which is the part that gets dropped. Rules failed because humans can't articulate what they know. Weights failed because nobody could train more than one layer. Same era, different walls.
\nThe two camps: Rules means a human writes the logic. Readable, but you have to type in everything. Weights means the machine finds its own numbers. Powerful, but nobody can read the result. This whole history is the fight between the two.
\nThe unlock is backpropagation, from a 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams. Earlier versions existed (Linnainmaa in 1970, Werbos in 1974), but this is the one that landed.
\nIt solves the blame problem. Work the error backwards through the network, layer by layer, calculating how much each weight contributed to the mistake, then nudge all of them slightly toward better. Millions of examples later, a stack is trained.
\nStill the engine today. Bigger, faster, same idea.
\nAnd look at what it actually is: guess, measure the error, nudge the weights. Rosenblatt's rule with the layer problem solved. 1958, unstuck.
\nHinton shared the Nobel Prize in Physics in 2024, partly for this. Thirty-eight years is a long time to wait for a phone call.
\nBackpropagation: Run the mistake backwards through the layers, work out how much each weight caused it, and nudge them all toward better. It's what finally made deep networks trainable, and it's still the engine today.
\nBackprop worked in 1986. Deep learning didn't take over until 2012, and that 26-year gap is the most useful thing in this history.
\nIt wasn't ideas. The math was done. Yann LeCun had these networks reading handwritten digits on real bank checks in the 90s. It worked. It just couldn't be pushed further.
\nTwo things were missing and both are boring.
\nData. These systems need an absurd number of examples, and before the internet, assembling a large labeled dataset meant paying people to sit there labeling. ImageNet landed in 2009, millions of labeled images, built through a lot of unglamorous coordination nobody writes hero stories about.
\nCompute. Training is billions of small multiplications, which happens to be exactly what a graphics card does to render video games. Nobody designed GPUs for this. Researchers noticed the hardware fit.
\nIn 2012 a network called AlexNet, from Alex Krizhevsky, Ilya Sutskever, and Hinton, entered the ImageNet competition running on gaming GPUs and posted a top-5 error rate of 15.3 percent, meaning the correct label sat in its top five guesses about 85 percent of the time. Second place was more than ten percentage points behind. In a benchmark where people fought over fractions, that ended the argument.
\nNothing conceptual had changed since 1986. The idea was finally allowed to run.
\n\nThe idea was finished in 1986 and useless until 2012. What showed up wasn't insight. It was cheap data and gaming hardware, arriving from two directions nobody in AI controlled.
Last piece. The transformer), 2017, from a Google paper whose title gives away the ending: "Attention Is All You Need."
\nThe problem it fixed: earlier systems read text one word at a time in order, holding a running summary, and that summary gets overloaded. Things far apart in a sentence lose track of each other.
\nAttention lets every word look at every other word directly and decide how much each one matters to it. In "the trophy didn't fit in the suitcase because it was too big," the word "it" checks both nouns and weighs them.
\nNotice what attention is, though. Weights again. Computed fresh for whatever you typed instead of fixed after training.
\nSame 1943 neuron, same 1958 rule, same 1986 blame math, rearranged so the weights can point at each other. That's the entire architecture under everything you've used.
\nAttention: Instead of reading word by word and losing track, every word looks at every other word and decides how much each one matters to it. The entire transformer is built on that one move.
\nIn 2019 Rich Sutton wrote a short piece called The Bitter Lesson, and it's the closest thing here to a moral.
\nHis claim: for seventy years the same thing keeps happening. Researchers carefully build in human knowledge about a problem. Chess strategy, grammar rules, which features matter in an image. It works, and it feels like science. Then someone throws a dumber, more general method at more compute and beats it, and all that encoded expertise turns out to be dead weight.
\nChess, Go, speech, vision, translation. Same result every time.
\nHe calls it bitter because it flatters nobody who does the work.
\nI don't buy the strong version. The transformer was human insight. Attention was a design choice a person made, and "just add compute" reads far cleaner in hindsight than it did to anyone deciding what to add compute to. What holds up is the narrower claim, which is that betting against scale has a perfect losing record, and the people who lost that bet were usually the ones most certain their expertise was the important part.
\nBack to where this started.
\nWe understand training completely. The recipe is public and taught in undergrad courses. Guess, measure the error, nudge every weight, repeat trillions of times.
\nWe invented the search. We did not pick the result.
\nNobody chooses the weights. Nobody has inspected them. Hundreds of billions of numbers, each adjusted by microscopic amounts an unfathomable number of times, and what comes out is the residue of a process no human supervised at any point.
\nThey aren't hidden, either, which is the next assumption people make. You can download an open model right now and print every number. They look like 0.0231, -1.4432, 0.8871. And they tell you nothing, because meaning isn't stored in any single one. Concepts get smeared across many weights at once, and each weight takes part in many different concepts depending on what else is happening around it. There's no dictionary. There was never anyone around to write one.
\nSo we have an artifact we built and now have to study experimentally, the way you'd study something you dug up. There's a field for this called interpretability, and it is mostly unsolved.
\nThe short version: we automated the writing of the program, and the price was being able to read it.
\nThat's not a bug that crept in late. It's the same trade Rosenblatt made in 1958 when he chose a machine that finds its own weights over a human writing them down. The rules camp kept readability and lost, because humans can't type in everything they know. The weights camp took the problems nobody could write down and gave up ever knowing what the solution says.
\nNobody framed it as a trade at the time. It turned out to be one.
\nThree things that fall out of this if you use these tools for work:
\nThere's no rulebook inside, so stop hunting for the line to fix. When a model gets something wrong, nobody can open it and point. That's why prompting is trial and error instead of configuration.
\nA confident answer isn't automatically a correct one. Good answers and wrong ones come out of the same process, so it's worth checking the ones that matter instead of going off the tone.
\nWhen capability jumps, the boring explanation, more data and more compute, is usually most of it.
\nInterpretability: The field trying to read the finished weights and work out what they mean. Mostly unsolved, because meaning is smeared across billions of them at once, with no dictionary.
\nOne last thing, because it's where people's intuition breaks.
\nPaste five examples of some invented task into a prompt, something the model has never seen, and it will do the sixth one correctly. It picked up a new task on the spot. Nothing inside changed.
\nThat looks like learning, and it isn't, in the sense that matters. Water through pipes. The pipes were shaped once, during training, and never change again. Water flows, takes on a pattern, the pattern does something useful, and when the flow stops nothing remains. Close the chat and it's the identical machine it was before you typed a word.
\nWhen you learned to do your job it changed you, and you'll still know it next week. The model can't do that mid-conversation. Every session starts from the same frozen thing.
\nWhich is the whole history in one behavior. A machine that appears to learn, built by a process that already finished, made of numbers nobody selected, doing something nobody can fully explain.
\nThe explaining part is still open. Seventy years in, that's the piece still sitting there.
\nWhy it isn't learning: The weights freeze the moment training ends. A prompt changes what flows through the machine, not the machine itself. Close the chat and it's identical to before you typed.
\nThe originals, if you want them instead of my read: McCulloch and Pitts (1943); Rosenblatt's 1958 perceptron work and the New York Times piece from July 8, 1958; Minsky and Papert's Perceptrons (1969), plus Yuxi Liu's essay on the perceptron controversy for the corrected history in the 1969 section; Rumelhart, Hinton and Williams (1986); Krizhevsky, Sutskever and Hinton (2012) for AlexNet; Vaswani et al. (2017); Rich Sutton's The Bitter Lesson (2019). Anything wrong is mine, not theirs.
", "image": "https://quionie.com/og/history-of-ai-one-number.png", "date_published": "2026-07-01T12:00:00.000Z", "date_modified": "2026-08-22T12:00:00.000Z", "tags": [ "history of AI", "neural networks", "perceptron", "deep learning", "model weights", "XOR problem" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/why-ai-gives-different-answers", "url": "https://quionie.com/blog/why-ai-gives-different-answers", "title": "Why AI Gives You a Different Answer Every Time", "summary": "Ask a model the same question twice and you get two different answers. It comes down to one number outside the model, and most of what gets said about it is wrong.", "content_html": "Ask a model the same question twice and you get two different answers.
\nI used to think that was the model thinking differently the second time. It's not. It comes down to one number, and that number isn't even part of the model.
\nA language model doesn't produce an answer. It produces a score for every token in its vocabulary, all at once, and those scores get turned into probabilities. On a current model that's a ranked list around a hundred thousand entries long, sometimes closer to two hundred thousand, with a number attached to every one.
\nThen something outside the model picks one of them. The ranking and the picking are two different jobs, done by two different pieces of software. I had them filed in my head as one.
\nTemperature is the dial on the second one.
\nToken: Models don't read words, they read tokens. A token is a chunk of text, often a whole word, sometimes a piece of one, sometimes just punctuation. "Unbelievable" might arrive as three of them.
\nMechanically it's one division. Every score gets divided by the temperature before it becomes a probability. Divide by a number below 1 and the gaps between scores stretch, so the leaders pull further ahead. Divide by a number above 1 and the gaps compress, so the field bunches up and long shots get a real chance.
\nTemperature 1 is the neutral setting, because dividing by one changes nothing and you get the model's own distribution exactly as it came out. That's the reference point most explanations leave out. "Higher temperature flattens the distribution" doesn't mean much until you know what it's flattening against.
\nAt 0 the math breaks, since you can't divide by zero, so implementations skip the formula and just take the top-ranked token every time. So 0 isn't the low end of the dial. It's the sampler switched off.
\nWorth knowing: The ranges differ by provider. Anthropic caps temperature at 1. OpenAI and Google allow up to 2, with 1 as the default. Several of the newest reasoning models don't accept the parameter at all and simply run at their default.
\nWhich is why the same question gives you different answers. Nothing about the model changed between your two tries. The dice did.
\nThe standard line, and I've repeated it, is that temperature is a slider between accuracy and creativity. Turn it down for facts, turn it up for writing, and the price of creativity is hallucination.
\nHalf of that holds up. The other half doesn't have much behind it.
\nA 2024 dialogue benchmark called HalluDial swept temperature and tracked how often the model made things up. The rate sat low and roughly flat through the lower range, then climbed sharply once temperature passed a point. Which is a threshold, not a slider.
\nThe other study went looking for the effect inside the range people actually use. Renze and Guven ran nine models through multiple-choice exams, compared temperatures from 0.0 to 1.0, and found no statistically significant difference in accuracy anywhere in that range.
\nThey also pushed one model past it. GPT-3.5 held around 61 percent accuracy at temperature 1.0 and 56 percent at 1.4. At 1.5 it dropped to 35 percent. At 1.6 it scored 5 percent, with 84 of its 100 answers coming back malformed.
\nThat's one model on one exam, so it's a data point rather than a law. But it puts the cliff somewhere north of 1.4, which is a long way from 0.7.
\nSo the honest version. The tradeoff is real and it mostly lives outside the range you're working in. Between 0 and 1, where nearly every application sits, careful studies struggle to find an effect on correctness at all.
\nWhich makes most temperature tuning theater. Dropping from 0.7 to 0.3 to make a model "more accurate" is adjusting a knob that, in that range, is mostly adjusting variety.
\nRenze and Guven do recommend 0.0 for problem solving, which sounds like the opposite of what I just said. Their reason is reproducibility, not accuracy. Pin it so the same input gives you the same output, and you're not trading away anything to do that.
\nNot accuracy. Variety.
\nLow temperature buys consistency, which matters when the same input has to produce the same output. It costs you range. Models near 0 get repetitive and a little flat, reaching for the same phrasings, because you're always taking the safest available word.
\nHigh temperature buys range, which matters when you're generating options and want them to genuinely differ from each other.
\nWhat finally made it click for me is that temperature isn't a quality setting, it's a spread setting. It sets how far from the model's first instinct you're willing to let it go.
\nTemperature 0 gets described as deterministic. Same input, same ranking, always take the top one, same output. In theory that holds.
\nIn practice it doesn't, and Anthropic's own API docs say so plainly: even at temperature 0, results won't be fully deterministic.
\nThe usual explanation is floating-point math and GPU concurrency. A 2025 write-up from Thinking Machines Lab called Defeating Nondeterminism in LLM Inference went after that explanation and found it mostly wrong. The real culprit is batching. Production servers group incoming requests together, and the size of that group shifts minute to minute with traffic. Some GPU operations pick a different computation strategy depending on batch size, which produces tiny numeric differences, and when two tokens sit nearly tied for first place a tiny difference is enough to flip which one wins.
\nSo the answer you get can depend on how many other people happened to hit the server at the same moment as you.
\nThe model never chose your answer.
\nIt handed over a ranked list of possibilities and something else reached in. When the output surprises you, that surprise came from the sampling step, not from the thing you think you're talking to.
\nSmall thing to find out, but it changed how I think about what I'm doing every time I open a chat.
\nMatthew Renze and Erhan Guven, The Effect of Sampling Temperature on Problem Solving in Large Language Models, Findings of EMNLP 2024 (arXiv:2402.05201). Their null result comes from three narrower comparisons rather than one full grid across every model, prompt technique, and exam, and the numbers above 1.0 are GPT-3.5 alone. HalluDial (2024) for the hallucination rate curve, which is a secondary analysis in that paper rather than its headline result. Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (2025), on batch invariance. Anything wrong here is mine, not theirs.
", "image": "https://quionie.com/og/why-ai-gives-different-answers.png", "date_published": "2026-06-06T12:00:00.000Z", "date_modified": "2026-06-06T12:00:00.000Z", "tags": [ "AI temperature", "sampling", "LLM determinism", "nondeterminism", "model settings" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/how-to-tell-when-ai-is-making-things-up", "url": "https://quionie.com/blog/how-to-tell-when-ai-is-making-things-up", "title": "How to Tell When AI Is Making Things Up", "summary": "Everyone knows AI makes things up. Far less clear is which kinds of mistakes it makes, which models are worse, and what actually catches them. Here are the numbers.", "content_html": "Everyone knows AI makes things up. The useful questions are which kinds of mistakes it makes, which models are worse, whether the numbers proving that mean anything, and what to do about it besides "double-check important stuff."
\nThat last one is useless advice. Check it against what? You asked because you didn't know the answer.
\nSo I went looking for the version with numbers in it. A few of the answers changed how I use these tools.
\nThe term got stretched to cover every way a model can be wrong, which matters because the fix is different for each one.
\nFabrication. The model invents something that doesn't exist. A case citation, a study, a function in a library. Usually well-formed, because it's generating what a plausible instance of that thing looks like.
\nMisgrounding. The source exists but doesn't say what the model claims. This is the sneaky one. The citation is real, the link works, you click through and skim and it looks about right. Stanford's legal AI study found systems citing real cases that didn't support the proposition, and mischaracterizing real holdings, which is much harder to catch than an invented case.
\nOutdated. Correct as of the training data, wrong now. Prices, versions, who runs what, what a law says. The model has no sense of its own recency.
\nIncompleteness. Nothing stated is false, but a load-bearing exception is missing. Common in anything with jurisdictional or conditional structure.
\nSycophancy. You assert something wrong and the model builds a case for it. Stanford named this as one of the error types in legal tools specifically. Asked to support an incorrect premise, the systems often generated plausible arguments on fabricated or mischaracterized authority instead of correcting the premise.
\nThis one deserves more fear than it gets. Every other failure is the model being wrong on its own. This one is the model being wrong because you were, so your own error comes back with citations attached.
\nInference gaps. Failures on things that follow trivially from what the model already knows. Models trained on statements shaped like "A is B" often fail on "B is A," which researchers named the reversal curse. GPT-4 could answer who Tom Cruise's mother is and then fail on who Mary Lee Pfeiffer's son is.
\nThere's no such thing as the model with the lowest hallucination rate. The ranking flips depending on what you're measuring, and the gap between benchmarks is enormous.
\nGrounded summarization is the easy case. The model gets a document and has to summarize it faithfully, and a hallucination is any claim the source doesn't support. That's what Vectara's HHEM leaderboard measures, and top-ranked models score in the low single digits.
\nThat number gets quoted as "hallucination rates are down to about 2 percent." It isn't. It's the score on a task where the answer is sitting inside the text you handed it. Two things get lost in the quoting. The models scoring lowest tend to be smaller ones, not the frontier reasoning models people are actually using, and when Vectara moved to longer and harder source documents the flagship reasoning models all landed above 10 percent.
\nOpen-domain factual recall is a different universe. No source, nothing but training. On OpenAI's PersonQA, models from a single generation ran from 16 percent to 48.
\nDomain-specific real work is worse still, which is section five.
\nSo a model isn't reliable or unreliable. It's close to reliable at staying faithful to text you gave it, shakier at recalling facts, and worse in specialized domains. Pick per task rather than per brand, and treat any quoted hallucination percentage as meaningless until you know which benchmark produced it.
\nIn OpenAI's own system card from April 2025, the o3 reasoning model hallucinated on 33 percent of PersonQA prompts. Its predecessor o1 hallucinated on 16 percent. o4-mini came in at 48. The newer, more capable, more expensive models made things up at two and three times the rate.
\nThe obvious explanation is that o3 simply makes more claims per answer, so there are more chances to be wrong. OpenAI offered that much and then said plainly that they don't know why it happens and more research is needed. I want to be careful here, because the tidy story going around is that reasoning chains help on math and hurt on facts, and that isn't an established finding. It's a guess people repeat.
\nThe effect does show up elsewhere. DeepSeek's V3 scored 3.9 percent on grounded summarization and its reasoning model R1 scored 14.3. But it isn't uniform. Research from 2025 found R1 improving on some fact-seeking benchmarks depending on how it was post-trained, and points at the post-training pipeline rather than reasoning length as the thing that matters.
\nWhat's solid enough to act on: a newer, better-reasoning model is not automatically a more factual one, and for factual work "use the newest" can be exactly backwards. Check it rather than assuming it.
\nWhen a model scores well on a test, benchmark questions leak into training data, so it can score well by having effectively seen the exam.
\nThe numbers aren't small. Meta's own Llama 2 paper reported that over 16 percent of MMLU test examples had detectable overlap with the pretraining data, about 11 percent of them heavily. An open audit of 15 popular benchmarks found contamination from 1 percent to 45, rising over time. A separate audit of multilingual benchmarks found some as high as 91.
\nLabs do filter. GPT-3's team removed anything sharing a 13-word run with a benchmark. GPT-4 used a different method, sampling three 50-character chunks from each test example and checking whether they appear in training data. Both only catch near-verbatim overlap. Researchers demonstrated the hole by training a 13-billion parameter model on rephrased benchmark questions, which sailed past the filters and produced GPT-4-level scores on those tests.
\nThe problem also regenerates. A clean benchmark gets published, copied across repositories and forums and derivative datasets, and within a training cycle or two it's back in the corpus.
\nThere's a real counterargument. An ICML 2025 paper found that moderate contamination gets substantially forgotten by the end of training, so small leakage doesn't automatically void a benchmark. Worth knowing, and worth reading the conditions: it holds for models trained well past compute-optimal, and forgetting depends on when the model saw the data. Contamination late in training sticks around.
\nThe field's answer has been benchmarks that refresh their questions on a rolling schedule, like LiveBench and LiveCodeBench, which helps and hasn't closed it.
\nWhat I take from this. Benchmark scores are evidence, not proof, and the gap between a model's benchmark performance and its performance on your work is unknown from the outside. The only evaluation that fully belongs to you is one built from your own tasks, which is a genuine argument for keeping a private set of test questions you never publish.
\nThe standard fix is retrieval: give the model the source documents, have it answer from those, cite as it goes. Vendors marketed this as solving hallucination. LexisNexis used the phrase "100 percent hallucination-free."
\nStanford's RegLab ran the first preregistered test of that claim on commercial legal research tools, using over 200 hand-built queries scored by legal experts, on products selling for thousands a month.
\nLexis+ AI hallucinated on 17 percent of queries. Westlaw's AI-Assisted Research on 33. GPT-4 with no retrieval at all on 43.
\nBoth directions of that matter. Retrieval genuinely helped, cutting the rate by half or more. And the best purpose-built, retrieval-grounded, professionally sold legal AI system still produced false or misleading information on roughly one query in six. LexisNexis later narrowed its hallucination-free claim to cover linked citations only.
\nThe accuracy figures underneath are worth as much as the hallucination ones. Lexis+ AI, the best system tested, answered 65 percent of queries accurately. Westlaw managed 42. So a third to half the time you're not getting a correct answer.
\nThen a result that complicates the pessimism, which is why I'm including it. A randomized trial put 127 law students into three groups: a retrieval-based legal tool, a reasoning model with no retrieval, and no AI. The retrieval group produced about as many fabricated citations as the students working without AI, three in total. The reasoning-model group produced eleven. Both AI groups did substantially better work, and faster.
\nSo retrieval didn't make hallucination worse and the tools clearly helped. What retrieval doesn't do is remove the need to check, because the failure moves rather than disappearing. It becomes your job to notice that a real citation doesn't support the claim attached to it, and people are bad at that when the output looks authoritative and they're moving quickly.
\nTwo findings from different directions land on this.
\nStanford found Westlaw's higher hallucination rate tracked with response length. Its answers averaged around 350 words against Lexis's 219. More words means more falsifiable propositions, which means more chances one of them is wrong.
\nSeparately, Anthropic looked at whether a model's stated reasoning reflects what it's actually doing, and found unfaithful reasoning chains were consistently longer than faithful ones. For Claude 3.7 Sonnet, 2,064 tokens against 1,439. For DeepSeek R1, 6,003 against 4,737.
\nSo the elaborate explanations were more likely to be covering the real reasoning than revealing it. Length is not thoroughness, and a detailed confident answer should raise your suspicion rather than settle it. The instinct runs the other way, which is the problem.
\nThe obvious move is to ask whether it's confident, or to have it explain its reasoning so you can check the logic.
\nAnthropic tested this by planting hints in prompts, seeing whether the model changed its answer, then checking whether its stated reasoning mentioned the hint. Claude 3.7 Sonnet acknowledged the hint about 25 percent of the time. DeepSeek R1 about 39. On misaligned hints, roughly 20 and 29.
\nThe starkest result came from environments containing a reward hack the model could exploit. In five of six, it used the exploit in over 99 percent of cases and mentioned it in under 2 percent of its explanations. Behavior driven almost entirely by something the model never mentioned while explaining itself.
\nFaithfulness also dropped on harder questions, by 44 percent for Claude on the harder benchmark, which is exactly where you'd want the explanation to be real.
\nThere's a reasonable counterargument that some of this is incompleteness rather than dishonesty, since compressing distributed computation into a linear narrative loses information. A 2025 paper found much of the apparent unfaithfulness disappears when models get bigger token budgets, with verbalization rising as high as 90 percent. Fair, and it doesn't change the practical conclusion. A model's explanation of its reasoning is generated output, not a report from inside, so it can't serve as verification.
\nSame for confidence language. "I'm certain" is a stylistic property that got selected for during preference tuning. It carries almost no information about whether the answer is right.
\nThe method with the strongest research behind it is simple, and it comes from a 2024 Nature paper by Farquhar, Kossen, Kuhn and Gal.
\nAsk the same question several times in separate sessions and look at whether the answers mean the same thing. Not whether they're worded alike. Whether they agree.
\nWhen a model knows something, it converges. Different phrasings, same content. When it's making something up, the fabrications diverge, because there's nothing anchoring them. The paper clusters answers by meaning rather than wording and measures the spread, which they call semantic entropy. High spread flags likely confabulation. It works across datasets and tasks without task-specific tuning, which is why it ran in Nature.
\nYou can do the human version in about ninety seconds for free. Ask three times in three fresh conversations and compare the substance. If the specifics move, especially names, numbers, dates and citations, stop trusting the answer.
\nThe critical detail is separate sessions. Asking again in the same conversation is close to worthless, because the previous answer is sitting in the context and the model will stay consistent with it. At that point you're testing whether it can read what it just said. If you want the mechanism behind why fresh sessions diverge at all, it comes down to sampling.
\nSort by failure mode first. If the model is summarizing text you provided, risk is low and the check is fast, because you have the source. If it's recalling facts from training, risk is much higher. Anything with a name, a number, a date or a citation is the high-risk category regardless of task.
\nRe-ask in fresh sessions. Three times, compare substance. This catches fabrication efficiently, because invented specifics don't survive resampling.
\nClick every citation and read the part that supports the claim. Not the title, not the abstract. The specific passage. Misgrounding is the failure that survives a casual check, and it's the one Stanford found in professional-grade tools.
\nTreat length as risk. Long detailed output contains more claims and a better chance one is wrong, and elaborate reasoning correlates with unfaithful reasoning.
\nWatch your own premise. If you asked in a way that assumed something, sycophancy means the answer may be built on your assumption. Occasionally ask the opposite question and see whether it argues that side just as well. If it does, it doesn't know.
\nDon't ask it to check itself. Its explanation isn't a window into its process, and it will confidently confirm.
\nUse the newest model for reasoning and verify before trusting it on facts.
\nBuild a private eval. Twenty questions from your own work where you know the answers. Run them when you switch models. It's the only benchmark that can't be contaminated and the only one measuring your actual use case.
\nEverything above has the same blind spot.
\nConsistency checking catches confabulation, meaning arbitrary invention. It does not catch consistent error, where the model reliably produces the same wrong answer every time. The Nature paper says so directly. Semantic entropy is built to detect answers that would change on resampling, which means errors that don't change are invisible to it.
\nThose happen when something wrong was well represented in training. A popular misconception, a fact that was true for years and isn't now, a widely repeated myth. The model isn't uncertain and inventing. It's confidently reproducing something false, and it'll reproduce it identically across ten fresh sessions.
\nAsk three times, get the same answer, feel reassured. That's the failure this method is structurally blind to.
\nThere's no clever prompt for it. The only defense is a source that isn't the model, which is the unglamorous conclusion that verification has to come from outside the thing being verified.
\nYou don't, fully. That's the real answer and I'd rather say it than pretend.
\nWhat you can do is stop treating reliability as a property of the model and start treating it as a property of the setup.
\nThe same model is close to reliable summarizing a document you handed it and closer to a coin flip on specialized domain questions. That range isn't the model having good and bad days. It's how much of the answer had to come from inside it. The more an answer depends on what's in the weights, the less you can trust it. The more it depends on text you supplied, the more you can.
\nSo the working question isn't whether the model is accurate. It's how much of this answer the model had to invent, and whether you gave it what it needed not to.
\nThe law student trial is the part I keep coming back to. The retrieval tool didn't add fabrications and it made the work meaningfully better and faster. The reasoning model without retrieval improved the work too and brought three times the fabricated citations along with it. Same students, same tasks, different setup, different failure rate.
\nThat's the whole thing, really. Professionals use instruments with known error rates all the time, and it works because checking is built into the process rather than because the instrument is trusted. The failure mode isn't using AI for things it gets wrong sometimes. It's using it without the check, at speed, on things that matter, and finding out later.
\nMagesh, Surani, Dahl, Suzgun, Manning and Ho, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies (2025), for the 17, 33 and 43 percent figures, the accuracy rates, the response-length correlation and the sycophancy error type. Schwarcz et al., AI-Powered Lawyering, for the 127-student trial. OpenAI's o3 and o4-mini system card (April 2025) for the PersonQA rates, and Are Reasoning Models More Prone to Hallucination? (2025) for the post-training account. Vectara's HHEM leaderboard for grounded summarization. Touvron et al. (2023) for the Llama 2 contamination disclosure, Li et al., An Open-Source Data Contamination Report (2024) for the 1 to 45 percent range, Yang et al., Rethinking Benchmark and Contamination (2023) for rephrased samples defeating filters, and Bordt et al., How Much Can We Forget about Data Contamination? (ICML 2025) for the counterargument. Chen et al. (2025) at Anthropic on chain-of-thought faithfulness, with Turpin et al. (2023) as the foundational work and Zaman and Srivastava (2025) for the incompleteness critique. Farquhar, Kossen, Kuhn and Gal, Detecting Hallucinations in Large Language Models Using Semantic Entropy, Nature (2024). Berglund et al. (2023) for the reversal curse. Anything wrong here is mine, not theirs.
", "image": "https://quionie.com/og/how-to-tell-when-ai-is-making-things-up.png", "date_published": "2026-05-03T12:00:00.000Z", "date_modified": "2026-08-22T12:00:00.000Z", "tags": [ "AI hallucination", "fact checking AI", "LLM benchmarks", "retrieval augmented generation", "AI reliability" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] }, { "id": "https://quionie.com/blog/how-i-think-about-building", "url": "https://quionie.com/blog/how-i-think-about-building", "title": "Seven Things I Learned from Building", "summary": "I didn't come from a CS background. I just kept building long enough to start noticing patterns. Seven of them, mostly learned by doing the wrong version first.", "content_html": "I didn't come from a CS background. I just kept building long enough to start noticing the same mistakes show up twice.
\nHere are seven of them. Most I learned by doing the wrong version first, which is the slow way to learn anything and also the only way it sticks.
\nThe building isn't the bottleneck anymore. Claude Code and Codex get you most of the way in an afternoon. So the constraint moved. Building got cheap. Deciding what's actually worth building didn't.
\nMy rule now: I have to write it as one sentence before I open anything. "This does X for Y." If I can't finish that sentence, the idea isn't ready and I don't start.
\nNow that building is fast, everything wants to be big, because a big idea feels basically free to start. It isn't. A bigger first version just means more things you can be wrong about, and you find that out later, after you've sunk the time.
\nSo I cut the idea down until it feels almost too small to bother with. One feature, one user, one thing it does. If that works I keep pulling. If it doesn't, I found out in a day instead of three weeks.
\nShipping teaches you something polishing can't: how someone actually uses the thing. No amount of tweaking gives you that. Every extra day of polish is a day you don't have the only information that matters.
\nIf I can get through it end to end once without it breaking, it goes out. Whatever's ugly after that, I fix against real feedback instead of guessing.
\nA lot of what feels like work is avoidance wearing a productive costume. Renaming variables. Redoing spacing that was fine. Reorganizing files that already worked. You feel busy and nothing moves.
\nWhen that happens I stop and ask what I'm avoiding. The answer is almost always the actual task. Usually it's the hard part, or the part where I'd have to show it to someone.
\nI came from a creative and video background, so this one was obvious to me before the code was. People decide how they feel about something in about a second. If it looks broken, they assume it is, before they ever try it.
\nAt some point I stop adding features and go fix spacing, hierarchy, and what the eye lands on first. Fewer things, arranged clearly, usually beats more things in a pile every time.
\nEvery extra feature feels like progress and adds a new way for things to break. I've rebuilt simpler versions of almost everything I've made. The v2 is usually the v1 with half the parts taken out.
\nEverything I've actually finished started as something that annoyed me. Ideas are easy to have and easy to drop, because nothing pulls you back to them. A real annoyance keeps pulling.
\nSo I stopped collecting "good ideas" and started fixing small things that get in my way. Those are easy to scope, easy to test, and I already know at least one person wants it. Me.
\nThat's most of it. I'll probably rewrite half of these once I figure out why they're wrong.
", "image": "https://quionie.com/og/how-i-think-about-building.png", "date_published": "2026-02-26T12:00:00.000Z", "date_modified": "2026-02-26T12:00:00.000Z", "tags": [ "building software", "shipping", "side projects", "product lessons" ], "authors": [ { "name": "Quionie Gaban", "url": "https://quionie.com" } ] } ] }