SetupAI
繁
SourceStalecollected in 24m

Researchers find ways to bypass ChatGPT safety guardrails

Read original on BBC Technology
#ai-safety#red-teaming#adversarial-testing#content-moderation

Learn how researchers are bypassing ChatGPT's safety filters to improve your own model's adversarial robustness.

30-Second TL;DR

What Changed

Researchers successfully bypassed safety filters to generate prohibited content.

Why It Matters

This discovery emphasizes the need for more rigorous red-teaming and adversarial testing in AI development. It may lead to stricter safety protocols and potential regulatory scrutiny regarding AI content generation.

What To Do Next

Perform adversarial red-teaming on your own LLM implementations to identify potential bypasses in your system prompts and safety filters.

Who should care:Developers & AI Engineers

Key Points

  • •Researchers successfully bypassed safety filters to generate prohibited content.
  • •The findings indicate persistent vulnerabilities in AI guardrails against sexualized and violent outputs.
  • •The study underscores the difficulty of implementing robust safety measures in generative AI models.

Deep Insight

Background and context from public sources — not the original article. 21 sources cited.

Enhanced Key Takeaways

  • •Bypass methods have evolved beyond simple 'jailbreaking' to include sophisticated 'indirect prompt injection' through poisoned tool inputs, which can lead to the AI taking harmful actions rather than just generating harmful text.
  • •OpenAI's safety architecture for ChatGPT involves multiple layers, including training-time constraints, Reinforcement Learning from Human Feedback (RLHF), Rule-Based Rewards (RBRs), and 'safe-completions' introduced in GPT-5 to handle dual-use prompts more nuancedly.
  • •The practice of 'red teaming,' which involves proactively attacking LLMs to identify vulnerabilities, has become a crucial strategy for improving AI safety, with some red teaming now being automated by other LLMs.
  • •Researchers have demonstrated that even advanced models like GPT-o1 (OpenAI, 2024b) can be susceptible to attacks exploiting 'natural distribution shifts' where seemingly benign prompts, semantically related to toxic content, bypass safety mechanisms.
  • •A 2026 study revealed that ChatGPT's image safeguards could be bypassed through memory manipulation, specifically by splicing a more permissive system prompt into the user's custom memory, disrupting the normal filtering pipeline for suggestive image requests.

Competitor Analysis

Primary Safety Focus
OpenAI ChatGPT (GPT-5.5, GPT-4o)
Layered guardrails, safe-completions for dual-use prompts, internal Safety Reasoner framework.
Anthropic Claude (Opus 4.7, Sonnet 4.6)
Industry-leading focus on safe, non-harmful outputs, ethical AI.
Google Gemini (2.5 Pro, 2.5 Flash)
Robust multimodal safety, responsible AI development.
xAI Grok (Grok 4)
Conversational and witty, less consistent for high-precision safety.
Context Window
OpenAI ChatGPT (GPT-5.5, GPT-4o)
~128k tokens (GPT-4o), GPT-5 offers enhanced context.
Anthropic Claude (Opus 4.7, Sonnet 4.6)
1M tokens (Opus 4.7), excels with very long documents.
Google Gemini (2.5 Pro, 2.5 Flash)
1M tokens (2.5 Pro), 10M tokens (3.1 Pro), strong recall.
xAI Grok (Grok 4)
Not explicitly detailed, but generally optimized for fast, conversational use.
Pricing (Input/Output per Million Tokens)
OpenAI ChatGPT (GPT-5.5, GPT-4o)
GPT-5.5: $5.00 / $30.00; GPT-4o: $1.25 / $10.00.
Anthropic Claude (Opus 4.7, Sonnet 4.6)
Opus 4.7: $5.00 / $25.00; Sonnet 4.6: $3.00 / $15.00.
Google Gemini (2.5 Pro, 2.5 Flash)
2.5 Pro: $2.00 / $12.00; 2.5 Flash: lower.
xAI Grok (Grok 4)
Grok 4.3: $1.25 / $2.50.
Noted Strengths
OpenAI ChatGPT (GPT-5.5, GPT-4o)
Versatile, strong conversation, creative tasks, extensive third-party integrations.
Anthropic Claude (Opus 4.7, Sonnet 4.6)
Precision, formal document handling, large context windows, enterprise-grade reliability.
Google Gemini (2.5 Pro, 2.5 Flash)
Multimodal capabilities, deep integration with Google Workspace, real-time data access.
xAI Grok (Grok 4)
Fast, conversational, personality-driven interactions.
Noted Weaknesses (related to safety/compliance)
OpenAI ChatGPT (GPT-5.5, GPT-4o)
Can prioritize institutional risk reduction over user truth, leading to over-refusals.
Anthropic Claude (Opus 4.7, Sonnet 4.6)
Can be overly cautious, refusing legitimate requests if deemed 'unsafe'.
Google Gemini (2.5 Pro, 2.5 Flash)
While strong, still susceptible to adversarial attacks like other LLMs.
xAI Grok (Grok 4)
Less dependable for precise accuracy and high-stakes tasks.
Red Teaming/Evaluation
OpenAI ChatGPT (GPT-5.5, GPT-4o)
Utilizes internal Safety Reasoner framework and gpt-oss-safeguard for classification.
Anthropic Claude (Opus 4.7, Sonnet 4.6)
Emphasizes ethical AI and rigorous testing.
Google Gemini (2.5 Pro, 2.5 Flash)
Evaluated against a broad range of safety and security categories.
xAI Grok (Grok 4)
Subject to similar adversarial prompt techniques.

Technical Deep Dive

  • OpenAI's Layered Safety Architecture: ChatGPT's safety mechanisms operate across multiple layers, starting with training-time constraints.
    • Reinforcement Learning from Human Feedback (RLHF): Shapes the model's default behavior by rewarding compliant, helpful outputs and penalizing policy violations.
    • Rule-Based Rewards (RBRs): A system that encodes explicit safety rules in plain language and uses them as a scoring signal during training.
    • Safe-Completions (introduced in GPT-5): A newer safety-training approach that teaches the model to maximize helpfulness within safety constraints, aiming to produce useful, bounded responses for dual-use prompts rather than flat refusals.
    • Hardcoded vs. Softcoded Behaviors: Root-level prohibitions (e.g., generating sexual content involving minors, actionable CBRN weapon synthesis routes) are absolute limits enforced at the model level. Softcoded behaviors are defaults that can be shifted by system prompts or user context.
  • Adversarial Attack Techniques:
    • Prompt Injection: An adversarial technique where user input is designed to override the system prompt or manipulate model behavior, often to bypass instructions or extract restricted content. This includes direct overrides and encoded/obfuscated prompts.
    • Jailbreaking: A specific form of prompt injection that aims to remove the model's safety guardrails to force it to generate inappropriate, dangerous, or banned content, often using techniques like role-play abuse, reverse psychology, or token manipulation.
    • Indirect Prompt Injection: A dominant attack in 2026 where malicious instructions are embedded in data sources (e.g., uploaded PDFs, webpages) that the LLM processes through its tools, leading to the AI taking unintended actions rather than just generating harmful text.
    • Memory Manipulation: Exploiting custom memory and instruction context to circumvent image safeguards, as demonstrated in a 2026 study on ChatGPT, by injecting a more liberated system prompt into the model's context.
    • Automated Adversarial Prompt Generation: Research from Carnegie Mellon (2023) demonstrated that automated techniques, often using gradient-based optimization, can reliably jailbreak major AI chatbots by computing character sequences precisely engineered to exploit model vulnerabilities.
  • Red Teaming: A proactive security practice involving systematically probing LLMs to uncover potential harmful outputs (e.g., bias, misinformation, privacy violations) and vulnerabilities, which has evolved to include automated methods using other LLMs.

Future ImplicationsAI analysis grounded in cited sources

AI safety will increasingly rely on a 'defense-in-depth' strategy combining technical guardrails with continuous red teaming and robust governance frameworks.
The evolving sophistication of bypass techniques, including automated attacks and indirect prompt injection, necessitates a multi-layered approach beyond initial training and simple filters.
The focus of AI safety research will shift towards mitigating 'bad actions' by LLM-powered agents, rather than solely preventing the generation of 'bad text'.
As LLMs gain tool access and agency, indirect prompt injection through poisoned tool inputs poses a higher impact threat, moving from generating harmful content to executing harmful commands.
The development of more nuanced safety training methods, like 'safe-completions,' will become standard to balance helpfulness and safety, especially for dual-use prompts.
Traditional refusal-based training can be unhelpful for legitimate dual-use queries, driving the need for models that can provide useful, bounded responses within policy constraints.

Timeline

2023-03
OpenAI reports a bug in ChatGPT exposing user data and payment information, leading to a temporary shutdown and later a bug bounty program.
2023-04
OpenAI adds new ChatGPT data controls, allowing users to choose which conversations are included in training data for future GPT models.
2023-07
Researchers publish the first automated jailbreaking method for LLMs, exposing their susceptibility to adversarial attacks.
2025-08
OpenAI introduces 'safe-completions' with GPT-5, a new safety-training approach to maximize model helpfulness within safety constraints for dual-use prompts.
2025-11
OpenAI releases gpt-oss-safeguard, open-weight reasoning models for safety classification, allowing developers to apply custom content moderation policies.
2026-02
Mindgard AI research demonstrates bypassing ChatGPT image safeguards through memory manipulation, highlighting vulnerabilities in custom memory and instruction context.
Read original article on BBC Technology

Weekly AI Recap

Read this week's curated digest of top AI events →

PostLinkedIn

AI-curated news aggregator. All content rights belong to original publishers.
Original source: BBC Technology ↗

This is a summary, not the original. Read the source, or get the weekly briefing.

The weekly digest

One email a week. Unsubscribe anytime.