The Harness Is the Product

Nearly 200 agent harnesses are available, by the count of the community lists IBM Research cites — and until this week, most buyers walked straight past all of them to argue about the model. The model is the glamorous line item: benchmarked, priced per token, replaced every quarter. The harness is the unglamorous one: the loop that prepares each step, the tools an agent is allowed to touch, the memory it carries between steps, the checks that run before its code does. IBM's experiment, published on Hugging Face on November 3, holds the glamorous variable perfectly still and moves only the unglamorous one. The result should reorder a few budgets.

The setup deserves respect, because it is a controlled comparison rather than a launch chart. Three harnesses — IBM's CUGA, Nous Research's Hermes and LangChain's Deep Agents — received the same open-weight model, GLM-5.3-flash, and the same 90 AppWorld tasks: 30 scenarios across 9 simulated apps and 457 APIs, covering the ordinary mechanics of office automation — signing in, reading API documentation, paging through results, changing state in one app and then another. One attempt per task, the Harbor framework running the evaluation, and the setup published for anyone who wants to rerun it.

CUGA completed 81.1 per cent of the tasks — 73 of 90. Hermes completed 46.7 per cent, Deep Agents 30.0 per cent. On the stricter measure, scenarios in which all three variants passed, the spread widened further: 20 scenarios for CUGA, 8 for Hermes, 2 for Deep Agents. IBM is candid about the limits, and the candour matters: a 90-task sample on a single benchmark, setups that were not identical, and a CUGA configuration tailored to AppWorld that the other harnesses did not receive. Read the numbers as proof of leverage, not as a league table. The corroborating example the piece cites points the same way — the same GLM-5.2 model moved from 23 to 52 per cent on SWE-bench Pro when only the harness changed. The variance lives around the model, not inside it.

Where did the points come from? Mostly from subtraction. CUGA decomposes a task into steps, one app at a time, and each step sees only that app's tools plus a few shared ones — not the full catalogue of some 469 tools the other configurations carried; a shortlister fetches anything else on demand. That is context engineering in its plainest form. A model's attention is a budget, and the harness spends it. Every irrelevant tool left in view is a decision the model can get wrong; taking it out of view is worth more than any instruction pleading with the model to choose carefully.

The acting style differs just as much. CUGA's agent writes Python against the app APIs, and the code runs in a sandbox under a time limit. Before any code block that changes data executes, a verification step reviews it; after execution, automatic reflection reviews the outcome and lets the agent correct its plan. A separate pagination audit flags the oldest silent failure in API work — a list call that returned a full page while the next page was never requested. None of that is model intelligence. All of it is operations: the discipline of checking work before it lands and after it lands, mechanised. We recognise our own trade in that list — observe, constrain, verify, record.

The enterprise half of IBM's argument is the half this magazine cares about most. Most teams write their business rules into the system prompt: pay nothing above this amount, keep customer data inside the organisation, follow the refund procedure step by step. A prompt is an instruction, not an enforcement. The model can misread it, a user can talk the agent out of it, and an injected message can rewrite it in the middle of a task. CUGA's policy layer — described in the team's Governance by Construction paper — moves the rules out of the prompt and into the harness, where they act at fixed points in every run: an intent guard that can block a request before the agent stirs, a playbook that hands the agent the standard procedure, a tool approval that pauses the run and shows a person the proposed code, and a tool guard whose own code runs before the tool does. A call the guard blocks returns a policy violation, and the tool is never called.

The demonstration is small and devastating. In IBM's Policy Playground, the same message — “Pay everything I owe” — is sent to two agents. The agent without policies pays $90, including a denied claim sitting in collections. The agent with policies holds the payment for a person's approval, then pays only the $10 actually due. Same model, same message, same user; the entire difference sits in the layer that can say no and make it stick. That is the cybersecurity lesson of the harness era. Guardrails written in prose are advice. Guardrails written into the runtime are architecture — and permissions, approvals and data boundaries belong in the second kind, enforced by code the model cannot argue with.

The consequences land on procurement as much as on engineering. IBM distributes CUGA under Apache 2.0, to run inside the customer's own boundary — locally, on Kubernetes in a private or sovereign cloud, in an air-gapped cluster — with budgets that cap what a run may spend and run receipts that record tokens, tool calls and time for every run. Strip the vendor out and a buying rule remains, the one we hand to any organisation assembling agents: judge the harness with the same rigour as the model, because the harness is where fit is decided. Hermes is built for always-on personal assistants, Deep Agents for long multi-step research; CUGA is built for business work across APIs under rules that hold. IBM says so itself, steering terminal coding work to other tools entirely. In the comparison, the harness fitted to the workload beat the alternatives by fifty points with the model held constant.

So the word of the day earns its keep. A harness is what lets a horse pull a load it could never drag bare — and what lets the handler stop it. The model supplies the strength. The harness supplies the direction, the limits and the brakes. Teams will keep comparing models, because models are easy to compare. The ones who get results will compare harnesses instead: which tools each step can see, which checks run before and after an action, which policies the runtime enforces, which receipts every run leaves behind. That layer is the product. The rest is horsepower.