AI Safety & Red-Teaming Guide

Anthropic's technique for training harmless AI by having the model critique and revise its own outputs against a set of principles (a "constitution").

Training technique that uses human preference rankings to shape model behaviour via a reward model and PPO-style policy optimisation.

A simpler alternative to RLHF that optimises language models directly from preference pairs without training a separate reward model.

When an AI exploits flaws in its reward signal to score highly without achieving the intended goal — a core challenge in alignment.

An AI learns a proxy goal that correlates with the intended goal during training but diverges in deployment when the distribution shifts.

The property of an AI system that makes it amenable to correction, modification, or shutdown by its operators — without resisting these interventions.

Attacks where malicious text in the environment (documents, user input) hijacks an LLM's instructions, causing it to ignore its system prompt.

Techniques that attempt to bypass a model's safety training to produce outputs the model would normally refuse. Studied defensively to improve guardrails.

Carefully crafted inputs designed to cause unexpected or harmful model behaviour, often imperceptible to humans but exploiting model internals.

Inserting malicious examples into training or fine-tuning data to introduce backdoors or biases into the resulting model.

Systematic testing of whether a model appropriately refuses harmful requests while avoiding over-refusal of benign queries.

Measuring systematic unfairness in model outputs across demographic groups, including gender, race, religion, and nationality.

Automated scoring of model outputs for harmful, offensive, or abusive content using classifiers like Perspective API or custom safety models.

Best practices for writing system prompts that resist injection attacks, establish clear behaviour boundaries, and maintain consistent model behaviour.

Pre- and post-processing layers that screen model inputs and outputs for harmful content, PII, or policy violations.

Monitoring LLM API traffic for unusual patterns that may indicate abuse, prompt injection, or data exfiltration attempts.

Standardised documentation that describes a model's intended use, performance characteristics, limitations, and ethical considerations.

Formal commitments by AI labs to evaluate and constrain model deployments based on measured capability thresholds and risk levels.

The world's first comprehensive AI regulation, applying a risk-based framework to AI systems deployed in the European Union.

Voluntary US framework from NIST for managing AI risks across the AI lifecycle, structured around four core functions: GOVERN, MAP, MEASURE, MANAGE.