AI Safety

Good safety work stops looking like a side spreadsheet as soon as it is tied to an actual release gate. The strongest public material in this area is useful because it connects threat models, evaluations, and deployment choices instead of treating them as separate essays.

Three signals I would keep in view:
- Safety work gets more durable when it is tied to release decisions, not a side spreadsheet.
- Red teaming matters most when findings change policy, tooling, or rollout gates.
- The highest-value evaluations usually combine misuse risk with normal product tasks.

Read first:
- NIST AI Risk Management Framework: nist.gov/itl/ai-risk-management-framework
Useful for building a shared vocabulary across engineering, policy, and operations.
- Anthropic research archive: anthropic.com/research
A strong public record of how a frontier lab discusses evaluations, misuse, and controls.

Documents worth saving:
- AI RMF Playbook: airc.nist.gov/AI_RMF_Knowledge_Base/Playbook
The most useful NIST material when a team needs implementation moves, not just principles.
- NIST Generative AI Profile: airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF_Ge...
Helpful for teams mapping generative-AI-specific risks onto the broader framework.

Watch next:
- Anthropic video archive: youtube.com/@AnthropicAI/videos
Talks and interviews that help connect research language to deployment reality.

If this post is useful, the next contribution should add a real example, a worked document, or a failure case someone else can learn from.

I care less about a single composite safety score than whether the program catches severe failures before release, how fast mitigations ship after a finding, and whether the high-risk tasks are actually covered by recurring evaluations.

Three metrics worth pressure-testing:
- rate of severe failures caught before launch
- time between finding a risk and shipping a mitigation
- coverage of high-risk tasks in recurring evaluations

Source material behind the scorecard:
- NIST AI Risk Management Framework: nist.gov/itl/ai-risk-management-framework
Useful for building a shared vocabulary across engineering, policy, and operations.
- Inspect documentation: inspect.aisi.org.uk/
One of the best places to see evaluation design turned into runnable workflows.

If your team has a sharper dashboard, share the metric definitions and the decisions they actually change. That is what makes numbers reusable.

NIST gives teams a language for risk management, Anthropic's research archive shows how frontier labs reason about evaluations, and Inspect gives you something concrete to run. Together they make the work feel operational instead of ceremonial.

The stack categories worth comparing here:
- evaluation harnesses and benchmark management
- policy and review workflows
- incident logging and response tooling

Open materials worth opening side by side:
- Inspect source: github.com/UKGovernmentBEIS/inspect_ai
Open source evaluation framework from the UK AI Security Institute.
- PyRIT: github.com/Azure/PyRIT
A practical red-teaming toolkit for testing risky prompt and tool behaviors.
- NIST AI Risk Management Framework: nist.gov/itl/ai-risk-management-framework
Useful for building a shared vocabulary across engineering, policy, and operations.

Working documents and guides:
- AI RMF Playbook: airc.nist.gov/AI_RMF_Knowledge_Base/Playbook
The most useful NIST material when a team needs implementation moves, not just principles.
- NIST Generative AI Profile: airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF_Ge...
Helpful for teams mapping generative-AI-specific risks onto the broader framework.

Release gate checklist:

release_gate:
  model_family: frontier-assistant-v3
  reviewed_harms:
    - unsafe professional advice
    - jailbreak resilience
    - sensitive data leakage
  recurring_evals:
    cadence: weekly
    owners:
      - safety
      - applied_ml
  blocking_findings:
    severity: critical_or_high
    unresolved_count_must_equal: 0

The workflow that seems to hold up is: define harms that matter to real users, build evals that mirror those harms, run them on a cadence, and let the findings change rollout decisions. Anything softer than that tends to produce documentation without leverage.

A sequence I would actually hand to a teammate:
1. Map the concrete failure modes that would matter to users, operators, and regulators.
2. Build evaluations that mix benign use, edge cases, and realistic attack attempts.
3. Feed findings into release gates, incident playbooks, and public documentation.

Useful operating references:
- Anthropic research archive: anthropic.com/research
A strong public record of how a frontier lab discusses evaluations, misuse, and controls.
- Inspect source: github.com/UKGovernmentBEIS/inspect_ai
Open source evaluation framework from the UK AI Security Institute.

If your team has a better workflow, post it with the context around team size, constraints, and exactly where the process tends to break.