Two agents can both return the correct label for a sentiment-classification task: one writes dozens of lines of Python, imports Transformers, hits a shape error, and re-runs twice; the other calls a one-line CLI. A traditional benchmark records both as identical successes, but what actually lands in your product cost is the process — how many turns, how many tokens, how many failures, and whether the official tool was used at all. Hugging Face’s June 18 post, “Is it agentic enough?”, targets exactly this blind spot: instead of asking whether the answer is right, it asks how much work the agent did to get there.
There is precedent. After Hugging Face reworked the hf CLI to be agent-optimized, agents used 1.3–1.8× (and up to 6×) fewer tokens. The effect is real, but verifying each tool change requires a repeatable measurement — so the team built the benchmark itself before shipping the changes.
Why the Final Answer Misses the Real Cost
Exact-match grading has two blind spots. First, large open models now saturate completion near 100% on common tasks, so match rates no longer separate them; the discriminating signal is effort — turns, tokens, wall-clock time, and whether the path was clean. Second, an agent that bypasses your official API and rewrites the logic from scratch still produces a correct answer, at a completely different cost. The benchmark therefore scores four axes: match rate (substring, regex, or exact), median time and tokens (with new, cached, and generated tracked separately), error rate (including a guard for silent failures — zero output and no tool calls), and marker adoption, which measures behavior rather than outcome.
Three Tiers Turn “Did the Docs Get Read?” into a Variable
Every task runs under three tiers, none of which contains another. Bare is pip install transformers and nothing else. Clone is a full source checkout. Skill packages the CLI’s documentation and task examples into the agent’s context. The tiers correspond to the three ways an agent can actually know your tool: training memory, reading source, or being handed the docs.
For now the harness only accepts deterministic tasks — answers that exact-match precisely; model-as-a-judge schemes are future work. That conservatism is deliberate: build the measurement foundation before extending to open-ended tasks.
How the Harness Works
The harness is a single CLI, agent-eval. You define tasks and expected answers, and it fans the models × revisions × tasks sweep out to Hugging Face Jobs — each run is an independent job for one (model × revision × task), and the whole sweep runs in parallel on identical hardware. It drives open models through the pi coding agent, and publishes the report as an HF Space with interactive visualizations and a shared-tasks-only toggle for like-for-like comparisons across models.
Markers are the mechanism that turns “what I want the agent to do” into a measurable pattern. Defined by a profile — a small per-tool plugin that teaches the harness how to build and drive a given library — a marker matches shell commands, code, files read, or the final answer. A cli marker, for instance, flags that the agent invoked the transformers CLI instead of handwriting Python.
Large Models: Less Time, More Tokens
The Transformers experiment fixes one strong model and varies the tool’s git revisions — v5.8.0, v5.9.0, up to the commit introducing the CLI and Skill. Across the three large open models Kimi-K2.6, GLM-5.1, and MiniMax-M2.7, the result is consistent: the Skill commit reduces time spent on tasks. Because the CLI landed in a single commit and is in no model’s training data, bare and clone runs almost never reach for it; on the skill tier, 55.3% of runs actually do.
The bill has two sides. On the clone tier, roughly a third of runs read the new interface first, and median input tokens rise from about 4k to 6.4k — less time, more tokens, a genuine trade-off.
Small Models: The Same Change Breaks Correctness
The small-model experiment inverts the design: hold the revision (with CLI + Skill) fixed and sweep across models, with size, quantization, and provider as variables. The story flips. Qwen3-4B reads the new CLI’s source heavily on the clone tier — median new tokens jump from about 2.4k to 23k — while time and output balloon and match rate gains nothing. Qwen3-14B fares worse: with the Skill, match drops from 67% on bare to 43%, classify-sentiment collapses from 100% on clone to 0%, and 39 of 56 skill runs mistake the CLI for a directly callable tool or simply give up.
The same change makes strong models faster and weak models worse. Without trajectory-level measurement, a maintainer could ship CLI + Skill believing it helps everyone, when in fact it adds friction for part of the audience.
What Tool Maintainers Should Take Away
The method is not Hugging Face-specific; any tool operable from a CLI can follow the same recipe. Define a few deterministic tasks, write a profile marking intended behavior, sweep the model sizes you intend to support, and read the match, token, error, and marker distributions. A few concrete judgments: validate API changes across model sizes, because benefits are not uniform — in this case the beneficiaries and the casualties were the largest and the smallest models; docs in context are not a tool, and weak models confuse the two, so documentation should explicitly direct “read, then invoke via shell”; and do not stop at match rate — silent failures and marker adoption often reveal interface problems earlier than wrong answers do.
Hugging Face also productized the conclusion: the companion project Upskill only generates a Skill when measurement shows it actually helps small models. One boundary before you run it yourself: the harness executes the coding agent with bypassed permissions, so run it only locally on trusted code.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.