🔍 Read the full analysis: Choosing An AI Harness: What Sets CUGA Apart From Hermes And Deep Agents on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The test suggests harness setup can affect results on this workload, but it does not establish a general ranking or measure policy enforcement.
IBM reports that its CUGA agent harness completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 tasks for Hermes and 27 for Deep Agents, as detailed in the original analysis. The comparison held the model and task set constant, offering evidence that harness design can shape performance on this particular multi-step benchmark—not that CUGA is generally better across agent workloads.
The experiment ran each of the three harnesses on the same 90 tasks across 30 scenarios, using GLM-5.3-flash and the Harbor evaluation framework. IBM says each task received one attempt. The reported completion rates were 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. On a stricter scenario measure—counting a scenario only if all three variants passed—CUGA passed 20 of 30 scenarios, Hermes eight and Deep Agents two.
The systems were not configured identically. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP alongside LangChain middleware for tool selection, context summarization, to-do lists, retries and call limits. IBM also says it made two fixes to Deep Agents so malformed model replies would not end a run.
IBM says the experiment setup is publicly available in harbor-eval, its fork of Harbor. The reported scores measure task completion under these configurations. They do not test every implementation difference independently, and they do not establish how the systems would perform with other models, workloads or repeated trials.
Harness Design Shapes Task Completion
The spread in the reported scores makes the harness a meaningful part of agent evaluation, not just a wrapper around a model. A harness shapes how an agent breaks down a task, selects tools, checks actions and responds to errors. When a job spans multiple apps or APIs, those orchestration choices can affect whether the task is completed.
For teams choosing an agent, the practical takeaway is to test the model, harness and workflow together. A benchmark score for a model alone may not predict how a configured agent will perform. At the same time, the results apply to one benchmark and do not show that CUGA will lead on the personal-assistant, research or other tasks associated with Hermes and Deep Agents.
The test also says nothing about whether an agent follows access rules or approval requirements. Those concerns matter for enterprise use, but policy enforcement was not evaluated in this experiment. IBM describes CUGA as having features for applying rules through its harness; the AppWorld scores do not establish how effective those features are.
Products Worth Considering
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
Samsung Odyssey G4 25-inch FHD Gaming Monitor
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Intended Uses
The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for work across enterprise systems such as customer relationship management and payments. It characterizes Hermes from Nous Research as aimed at always-on personal assistants, while Deep Agents from LangChain is presented as suited to longer, multi-step tasks such as research. Those descriptions help explain why a single benchmark should not be treated as a universal product contest.
AppWorld evaluates multi-step tasks across nine consumer apps and 457 APIs, including email, payments and file storage. IBM presents it as a proxy for some capabilities relevant to enterprise automation, such as reading API documentation, paging through results and changing information across apps. But the benchmark’s consumer-app environment does not directly establish performance on real enterprise systems.
The source also points to community directories that track nearly 200 agent harnesses and cites a separate directory analysis in which GLM-5.2’s SWE-bench Pro score rose from 23% to 52% when the harness changed. That is a distinct reported comparison, not part of the AppWorld test and not independent confirmation of its results.
“This is not a verdict on Hermes or Deep Agents.”
— IBM, describing the AppWorld results
Limits of the AppWorld Evidence
The results cover one model, one 90-task sample and one attempt per task. It remains unclear whether the relative scores would hold with different models, task sets, repeated runs or production workloads. The configurations also differed, so the comparison does not isolate the effect of one harness feature at a time.
IBM’s report, as described in the source material, does not provide comparative results for cost, latency, reliability at larger scale or deployment outcomes. The benchmark did not use policies, so it cannot establish whether the agents enforce access controls, approval steps or data boundaries. The source gives no date for an independent replication or a broader comparative study.
Test the Workflows You Need
The experiment’s setup is available through harbor-eval, according to IBM. Organizations considering these harnesses can use the reported comparison as a reason to run their own evaluations, rather than assume the benchmark predicts their results. Tests should use representative workflows, the intended model and tool configuration, and repeated or varied tasks to measure both completion and consistency.
For enterprise adoption, those evaluations would also need to cover access controls, approvals, data boundaries and operating conditions that AppWorld did not assess. The supplied material does not identify a scheduled follow-up study, so the timing of further comparative evidence remains unknown.
Key Questions
What did IBM report in the AppWorld test?
IBM reported that CUGA completed 73 of 90 tasks with GLM-5.3-flash. Hermes completed 42 and Deep Agents completed 27 in the same reported evaluation.
Does the result prove CUGA is the best AI harness?
No. The comparison covered one model, one benchmark and one task sample, with differently configured harnesses. It does not establish a general ranking across other uses.
What is AppWorld testing?
AppWorld tests multi-step tasks across nine consumer apps and 457 APIs, including email, payments and file storage. IBM describes it as a proxy for some capabilities relevant to enterprise automation, not a direct test of real enterprise systems.
Did the experiment test security policies or access controls?
No. The source says the experiment did not use policies, so its scores provide no evidence about policy enforcement, approval requirements or data boundaries.
What should organizations do with the results?
Use them as a reason to evaluate the specific model-and-harness combination on representative workflows. Teams should also test consistency, controls and deployment conditions that the benchmark did not cover.
Primary source: Hugging Face · via ThorstenMeyerAI.com
