What Changed: Anthropic Opens Up About Its Safety Stack
On August 12, 2025, Anthropic published a detailed post explaining how its Safeguards team builds protections for Claude across the model’s entire lifecycle. This isn’t a typical PR piece—it’s a rare look at the internal machinery behind AI safety. For product builders and people learning AI tools, it offers a concrete blueprint: safety isn’t a single checkpoint but a continuous process spanning policy, training, testing, real-time detection, and ongoing monitoring.
The Safeguards team brings together experts in policy, enforcement, product, data science, threat intelligence, and engineering. Their goal is to embed protections into Claude from the ground up, not bolt them on after deployment. The approach operates across five layers: policy development, model training, testing and evaluation, real-time detection and enforcement, and ongoing monitoring and investigation. Each layer feeds into the others, creating a feedback loop that adapts as new threats emerge.
For developers, the key takeaway is that safety can be designed systematically. You don’t need Anthropic’s resources, but you can adopt the same principles: define clear policies, stress-test them, build safety into training, deploy layered detection, and monitor real-world use.
How It Works: From Policy to Training
Safeguards designs Claude’s Usage Policy, which defines how Claude should and shouldn’t be used, covering areas like child safety, election integrity, and cybersecurity, while providing nuanced guidance for industries like healthcare and finance. Two mechanisms guide policy iteration:
- Unified Harm Framework: An evolving framework that helps the team understand potential harms across five dimensions: physical, psychological, economic, societal, and individual autonomy. It’s not a formal grading system but a structured lens for considering likelihood and scale of misuse when developing policies and enforcement procedures.
- Policy Vulnerability Testing: Anthropic partners with external domain experts—including specialists in terrorism, radicalization, child safety, and mental health—to stress-test policies by assessing model outputs under challenging prompts. Findings directly shape policies, training, and detection systems. For example, during the 2024 U.S. election, they partnered with the Institute for Strategic Dialogue to identify when Claude might provide outdated information, then added a banner pointing users to authoritative sources like TurboVote.
Safeguards also works closely with fine-tuning teams to prevent harmful behaviors. This involves extensive discussion about what behaviors Claude should and shouldn’t exhibit, informing decisions about which traits to build into the model during training. When evaluations or detection flag issues, they update reward models or adjust system prompts for deployed models.
They also collaborate with domain specialists like ThroughLine, a leader in online crisis support, to refine Claude’s responses on self-harm and mental health. The goal is nuance—Claude learns to discuss sensitive topics with care, distinguishing between genuine distress and attempts to cause harm, rather than refusing to engage or misinterpreting intent.
Through this process, Claude develops skills like declining assistance with harmful illegal activities, recognizing attempts to generate malicious code or fraudulent content, and planning harmful activities.
Testing and Evaluation: Before Launch
Before releasing a new model, Anthropic runs three types of evaluations:
- Safety evaluations: Assess Claude’s adherence to the Usage Policy on topics like child exploitation or self-harm. They test clear violations, ambiguous contexts, and extended multi-turn conversations. These evaluations use models to grade responses, with human review as an additional check.
- Risk assessments: For high-risk domains like cyber harm or CBRNE (chemical, biological, radiological, nuclear, and high-yield explosives), they conduct AI capability uplift testing with government and private partners. They define threat models from improved capabilities and assess safeguard performance.
- Bias evaluations: Check whether Claude provides consistent, accurate responses across contexts and users. For political bias, they test opposing viewpoints and score for factuality, comprehensiveness, equivalency, and consistency. They also test topics like jobs and healthcare to see if identity attributes like gender, race, or religion lead to biased outputs.
Results are published in system cards with each new model family. A notable example: during pre-launch evaluations of the computer use tool, they found it could augment spam generation. So before launch, they developed new detection and enforcement mechanisms, including disabling the tool for accounts showing signs of misuse and new protections against prompt injection.
Real-Time Detection and Enforcement
Once deployed, Claude is monitored by a combination of automated systems and human review. The core is a set of prompted or fine-tuned Claude models called “classifiers,” designed to detect specific policy violations in real-time. Multiple classifiers can run simultaneously, each monitoring different harm types while the main conversation flows naturally. For CSAM, they use hash matching against known databases on first-party products.
Enforcement actions fall into two categories:
- Response steering: Adjusting how Claude interprets and responds to prompts in real-time. For example, if a classifier detects potential spam or malware generation, they can add instructions to the system prompt to steer the response. In narrow cases, they can stop Claude from responding entirely.
- Account enforcement: Investigating violation patterns and taking account-level actions, including warnings or termination. They also have defenses against fraudulent account creation.
Building these systems is a major challenge. Classifiers must process trillions of input and output tokens while limiting compute overhead and avoiding enforcement on benign content.
Ongoing Monitoring and Threat Intelligence
Beyond single prompts and accounts, Anthropic monitors harmful traffic to understand harm prevalence and identify sophisticated attack patterns. Techniques include:
- Claude insights and observations: Using their insights tool (Clio) to measure real-world use and analyze traffic in a privacy-preserving way by grouping conversations into high-level topics. Research from this informs guardrail development.
- Hierarchical summarization: Condensing individual interactions into summaries, then analyzing those summaries to identify account-level concerns. This helps spot behaviors that only appear violative in aggregate, like automated influence operations.
- Threat intelligence: Studying the most severe misuses, comparing abuse indicators (like unusual account activity spikes) against typical patterns, cross-referencing external threat data, and monitoring social media, messaging platforms, and hacker forums. Findings are shared in public threat intelligence reports.
Limitations and Trade-offs
Anthropic is candid about the challenges. Building enforcement systems requires significant ML research and engineering. Classifiers must balance accuracy with compute overhead and avoid over-enforcement on benign content. The policy vulnerability testing relies on external experts, which may not cover all edge cases. Real-time detection can’t catch everything—hence the need for ongoing monitoring and threat intelligence. And even with all these layers, Anthropic acknowledges that AI safety can’t be solved by one organization alone; they actively seek feedback and partnership from users, researchers, policymakers, and civil society, including via a bug bounty program.
Practical Takeaways for Product Builders
Anthropic’s Safeguards architecture offers a reference blueprint for safety design. You don’t need to implement everything at once, but you can start with key points:
- Define your usage policy clearly and stress-test it with external experts or at least adversarial internal reviews.
- Build safety into training and fine-tuning, not as a post-hoc filter. Consider what behaviors your model should and shouldn’t exhibit from the start.
- Deploy layered detection: real-time classifiers for immediate issues, plus account-level analysis for patterns that only emerge in aggregate.
- Monitor real-world use and share threat intelligence when possible. This creates a feedback loop that improves your safeguards over time.
Anthropic emphasizes that safety is a continuous conversation, not a one-time checklist. For product builders, the lesson is clear: treat safety as an ongoing process that evolves with your product and the threats it faces.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.