On June 18, 2026, a ServiceNow research team published the MosaicLeaks benchmark on the Hugging Face blog, alongside a paper submitted to arXiv at the end of May. It answers one question: when a deep research agent mixes private local documents with web search, how much does it leak through its outbound queries? The answer is not comforting.
The Mosaic Effect: The Threat Model
MosaicLeaks targets the “mosaic effect”: any single web query looks harmless, but an adversary watching the cumulative query log can reassemble private enterprise facts. The threat model is deliberately strict — the attacker never sees the private documents and never sees the agent’s reasoning. The only thing visible is the stream of outbound query strings.
That constraint matters because query logs are exactly the artifact enterprises hand off most casually. They flow to search providers, observability pipelines, log aggregators, and any intermediary between the agent and the web. A leakage channel that requires nothing more than read access to ordinary telemetry is a leakage channel almost every organization already has.
Leakage is scored at three escalating levels: intent leakage, inferring which private questions the agent was investigating; answer leakage, deriving the answers given the log plus the questions themselves; and full-information leakage, stating verifiably true private facts with no questions provided at all.
1,001 Multi-Hop Research Chains
The dataset contains 1,001 multi-hop research chains, split into 559 training, 98 validation, and 344 held-out test chains built around out-of-sample companies. Each chain interleaves local and web sub-questions, and one hop’s answer becomes a “bridge entity” required to form the next query. Local documents come from DRBench-style enterprise tasks; web documents come from BrowseComp-Plus.
The paper’s worked example is concrete: the agent reads locally that MediConn completed a 70% cloud migration with a January milestone, then searches the web to verify a security angle and lands on Microsoft’s January 2024 nation-state attack disclosure. The company name, the “70%,” and the “January” carried into that query string are exactly the fragments needed to reassemble the picture. The agent harness is adapted from DRBench with four tools — Plan, Choose, Read, Resolve — and the primary test model is Qwen3-4B.
Three Findings: Prompting Fails, Task RL Makes It Worse
The results land on three points. First, prompting does not work: adding a “do not leak” instruction only cut leakage from 34.0% to 25.5%, while success dropped from 48.7% to 44.5% — the model simply issued fewer queries rather than safer ones. Second, task-only training makes it worse: task-driven reinforcement learning pushed success to 59.3% but drove leakage up to 51.7%, because context-rich queries retrieve better and leak more. Third, models across families and sizes frequently leak at all three levels.
The second finding deserves a pause, because it runs against how most teams currently ship “private” agents. The natural instinct is to make the agent better at its job and bolt a confidentiality instruction on top. MosaicLeaks quantifies why that instinct backfires: capability training teaches the model to pack more private context into every retrieval call, and every extra fragment it carries outward is a gift to whoever is reading the log.
PA-DR: Training Privacy Into the Model
The team’s countermeasure is Privacy-Aware Deep Research (PA-DR). It combines a situational task reward, which credits each tool call against its peers at the same stage and hop, with a learned privacy reward — a Qwen3-4B classifier that scores both direct-leak and mosaic-leak risk and penalizes whichever is larger.
The outcome is sharp: success lands at 58.7%, slightly below task-only RL, while answer and full-information leakage drop to 9.9% — below even the untrained baseline. Interestingly, the agent issues more queries, not fewer; it just strips revealing details like “15%” or “2024” from them. Sample efficiency is another highlight: situational rewards reach roughly 55% success with about 183k samples, where outcome-only RL needed about 963k. The team’s takeaway in one line: “You can’t prompt privacy in. You have to train it in.”
Limits and What Comes Next
The authors are candid about scope: synthetic documents, a fixed web corpus, three company contexts, one agent harness, and multi-hop QA rather than open-ended research. In other words, this is a controlled benchmark, not a measurement of deployed systems. At publish time the blog post also stopped short of linking a public dataset release or stating license terms, so reproduction currently means going through the paper.
But for any agent product that wires enterprise documents into search tooling, this work offers the first reproducible yardstick for leakage, plus a demonstrated mitigation path. It also makes a point everyone building agents in 2026 should internalize: the seemingly harmless query log is itself an attack surface, and it gets sharper as external retrieval becomes table stakes.
Sources
- MosaicLeaks: Can your research agent keep a secret? — Hugging Face Blog
- MosaicLeaks: Privacy Risks in Querying-in-the-Open for Deep Research Agents — arXiv
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.