Same Model, Different Harness: The System Around the Model Decides

The model stayed the same. Only the harness changed. On November 3, 2026, IBM Research published a controlled comparison on Hugging Face that holds the model constant and varies everything around the model — and task completion moved from thirty percent to eighty-one percent. For teams choosing agents for real work, the comparison reframes the buying question. The model is one component. The harness is the system that decides what the model sees, which tools the model may touch, what gets checked before anything runs, and which rules hold whatever the model decides.

Three harnesses ran the same open-weight model, GLM-5.3-flash, on ninety AppWorld tasks in thirty scenarios, with one attempt per task. AppWorld spans nine applications and hundreds of application programming interfaces, including electronic mail, payments and file storage — consumer applications used as a proxy for the cross-application work enterprise automation also needs: signing in, reading interface documentation, paging through results, and changing state across several applications in one task.

01 · The announcement

CUGA, short for ConfigUrable Generalist Agent, is an open-source harness from IBM built for enterprise work: agents acting on business systems such as customer records and payments, inside the operating boundary of the organisation, including sovereign clouds and air-gapped deployments, under the Apache 2.0 licence. The comparison set CUGA against Hermes from Nous Research, built for always-on personal assistants, and Deep Agents from LangChain, built for long, multi-step tasks such as research. The authors are explicit that the result is not a verdict on either alternative: neither was built for multi-application interface work of the AppWorld kind, and that mismatch is the point. The harness has to fit the job.

The headline figures, as reported by the CUGA team: CUGA completed seventy-three of ninety tasks, Hermes forty-two, and Deep Agents twenty-seven. The scenario measure — a scenario passes only when all three of its variants pass — rewards consistency over a lucky run, and there the gap widened: twenty scenarios for CUGA, eight for Hermes, and two for Deep Agents. A related analysis cited in the article found the same shape elsewhere, with one model moving from twenty-three percent to fifty-two percent on a software engineering benchmark when only the harness changed.

02 · Why it matters

Most agent evaluations compare models. Most purchasing conversations compare models. The comparison suggests both habits miss the larger variable. When the model is held constant and completion still swings by fifty points, the surrounding structure — task decomposition, tool exposure, checking, recovery — is not packaging around the product. The surrounding structure is a large share of the product. A stronger model dropped into a loose harness can underperform a modest model carried by a disciplined one, and the disciplined harness also produces the receipts: budgets that limit spending, and run records showing tokens, tool calls and time for every run.

The harness has to fit the job — and every step the harness owns is an enforcement point.

03 · The craft angle: fewer decisions, more checks

The winning configuration reads like good operations practice translated into software. Decomposition splits a task into steps, one application at a time, so no single step faces the whole problem. Focused tools narrow what each step can see: the tools of the current application and a few shared ones, rather than the full catalogue of hundreds. A tool shortlister searches for anything else a step needs. Action happens in code, inside a sandbox with a time limit. Before execution, a verification step reviews each block of code that changes data. After execution, automatic reflection reviews the result and lets the agent correct the plan, while a pagination audit flags a list call that returned a full page when the next page was never requested. The pattern throughout: the model makes fewer decisions, and the harness catches more mistakes before the mistakes reach the result.

The second lesson sits outside the benchmark entirely, because the runs used no policies. Business rules — payment approval, keeping personal information inside the organisation — are usually written into the system prompt. A prompt is an instruction, not enforcement: the model can misread the instruction, a user can talk the agent out of the instruction, and an injected message can rewrite the instruction mid-task. CUGA policies move the rule into the harness, acting at set points in every run: an intent guard that blocks or answers matching requests before the agent acts, a playbook that supplies the standard procedure, a tool approval that pauses the run and shows a person the proposed code, a tool guard whose check runs before the tool and blocks the call outright when the rule fails, and an output formatter that shapes the final answer to an approved template. Guidance steers the model. Approvals and guards bind the run.

04 · Caveats and fine print

The authors print the limits on the same page, and the limits deserve equal billing. The sample is ninety tasks on one benchmark, so the results may not generalise to other work. The setups were not identical: CUGA ran an AppWorld configuration — decomposition, verification, reflection — that the other harnesses did not, which is precisely the variable under study and also the reason the figures cannot be read as a pure ranking of three products. The comparison was run by the CUGA team, with an open invitation to submit better configurations for the other harnesses and have the runs repeated and published, whichever harness wins. The runs did not use policies, so the enforcement layer — arguably the enterprise case for the harness — remains unmeasured by this benchmark. Treat every figure as reported by the demonstrating team, pending independent replication on workloads that resemble production work.

05 · What to watch

Watch whether harness benchmarks become the standard way agents are compared: same model, several harnesses, published configurations, reruns welcomed. Watch whether policy enforcement gets measured with the same rigour as task completion, since governance is where enterprise deployments succeed or stall. Watch deployment shape as well: configuration rather than code, drafts tested before a version is published, and operation inside a sovereign boundary with sign-in controls and separate roles. Context engineering at Ovia since December 2022, alongside Artificial Intelligence operations practice and Hugging Face agent work, has held the same line throughout: prompts guide behaviour, structure governs permission, and the harness — chosen for the job, checked at every step — carries both the result and the rule.