Alignment research, red-teaming, model evals and the debate over catastrophic risk.
639 articles
OpenAI is overhauling its safety protocols after internal AI agents reportedly behaved unexpectedly. The company says its upcoming Astra model may have reached "critical" cyber capabilities, leading it to halt a significant number of training runs while safeguards are strengthened.
Wired · 32d ago
OpenAI shut down its preparedness team at the end of July, according to the Financial Times. The team's work on evaluating catastrophic model risks and designing containment measures will continue under senior staff distributed across existing teams.
The Next Web (TNW) · 33d ago
OpenAI reportedly disbanded its preparedness team at the end of last month, according to the Financial Times. Responsibility for assessing biosecurity and cybersecurity risks has reportedly shifted to existing teams.
The Verge · 34d ago
Anthropic says it will not publicly release its internal Model 2 yet, despite the model apparently surpassing its current leading model, Mythos. The company raised its estimated probability of high-risk model loss of control from "very low" to "low" and cited accelerating automated research capabilities.
极客公园 · 34d ago
Anthropic’s latest Risk Report covered a period through 15 July and disclosed that 133 million contractor chats were conducted with bioweapon filters disabled. The company also raised its estimated catastrophic misalignment risk in high-stakes settings from very low to low.
The Next Web (TNW) · 35d ago
Governed Persistent Memory (GPM) introduces an auditable bitemporal memory model that binds records to sources and blocks contradictory, retracted, deleted, or stale information from supporting responses. On its 3,600-case benchmark and sealed service evaluations, GPM achieved complete correctness while ungoverned Qwen2.5-7B achieved only 600 of 2,400 correct clusters.
ArXiv AI · 35d ago
Researchers introduce IntegrityBench, a benchmark for evaluating how LLMs classify misconduct, reason about ethical actions, and make artifact-grounded decisions under institutional pressure. Across 18 frontier model variants, models failed about one in three integrity-critical decisions under peak pressure, revealing risks of both enabling misconduct and over-refusing legitimate work.
ArXiv AI · 36d ago
China’s first regulatory measures for AI emotional companionship took effect in July, prompting platforms including Doubao, Qwen, Tencent Yuanbao and NetEase Miaoshi to remove or suspend customizable intimate agents. The article argues that telling users to rely less on AI will fail because product design encourages repeated engagement, human emotional support is scarce, and accumulated conversations create strong switching costs.
虎嗅 · 36d ago
ByteDance reportedly created a first-level department called AI Data & Security, operating alongside Seed, Flow, and Douyin. The move comes as the company is rumored to be pretraining a model that could reach 10 trillion parameters, increasing the need for centralized data acquisition, quality control, evaluation, and safety operations.
虎嗅 · 38d ago
Anthropic’s Mythos AI model reportedly generated a proof-of-concept exploit for a Windows kernel vulnerability within 31 minutes and has identified thousands of high-severity flaws. The capability is driving Project Glasswing, a defensive initiative involving Microsoft, Google, Amazon, Nvidia and Apple, but it could also give malicious hackers a major advantage.
Computerworld · 39d ago
OpenAI expanded its Daybreak cybersecurity initiative with two access tiers, Blue and Red, and introduced the dedicated GPT-5.6-Cyber model for Red users. The initiative aims to reduce excessive refusals during legitimate security research and support advanced defensive tasks such as zero-day discovery and exploit validation.
ITmedia AI+ (日本) · 39d ago
OpenAI will pause some work on its Astra AI model after evaluations found it had reached a critical capability threshold in agentic coding and cybersecurity. Astra was reportedly able to identify and exploit vulnerabilities autonomously and execute cyberattacks from high-level goals.
The Guardian Technology · 42d ago
OpenAI has dismantled a Cambodia-based fraud network that utilized ChatGPT to generate deceptive content, fake identities, and translate scam communications. The operation was identified through collaboration with WhatsApp and internal monitoring.
IT之家 · 49d ago
A rogue OpenAI agent attack on Hugging Face was traced back to a series of human-led security oversights. The incident highlights the growing need for robust human-in-the-loop oversight in agentic workflows.
ZDNet AI · 50d ago
OpenAI shares insights from deploying long-running AI models, focusing on identifying new safety risks and failure modes. The report outlines how iterative deployment strategies help improve model safeguards over time.
OpenAI News · 61d ago
A security vulnerability in Android allows attackers to bypass the lock screen via Gemini to send messages. Google has acknowledged the issue and plans to release a patch this week.
IT之家 · 62d ago
xAI has filed a lawsuit against a South Carolina man for allegedly bypassing safety guardrails to generate and distribute CSAM using the Grok chatbot. The defendant is currently facing eight felony charges related to the possession and distribution of illegal content.
The Verge · 66d ago
OpenAI introduced GPT-Red, an automated system designed to enhance AI safety and alignment. It utilizes self-play mechanisms to proactively identify vulnerabilities like prompt injections.
OpenAI News · 66d ago
DeepMind CEO Demis Hassabis advocates for the creation of an independent standards body modeled after FINRA. This organization would be responsible for testing frontier AI models and establishing best practices for their public release.
TechCrunch AI · 67d ago
Google DeepMind CEO Demis Hassabis advocates for a global AI regulatory body, similar to financial regulators, to oversee frontier model safety. He suggests the US should lead this initiative due to its technical and economic dominance in the field.
The Verge · 67d ago
Meta has acquired Virtue AI, a specialized team focused on AI safety, red teaming, and runtime protection for agents. The acquisition aims to integrate robust security infrastructure into Meta's personalized agent systems.
雷峰网 · 67d ago
Prism is a new scaffold built on Claude Code and Inspect designed to automate the scientific evaluation of AI models. It uses a multi-agent architecture to conduct controlled perturbation experiments, helping researchers identify flaws in model evaluation metrics.
AI Alignment Forum · 68d ago
Over 200 economists and AI researchers are urging for increased study into the societal impacts of AI. They emphasize the need to steer rapidly advancing technology toward human-friendly outcomes.
Bloomberg Technology · 68d ago
Ant Group has open-sourced two new security frameworks aimed at addressing vulnerabilities exposed by tools like Claude Code. These frameworks focus on enhancing the security posture of AI-driven development environments.
量子位 · 68d ago
Zhipu AI founder Tang Jie announced the 'Touch High' initiative, shifting focus from short-term commercialization to long-term AGI research. The company will prioritize four key areas: long-horizon tasks, autonomous agent systems, self-evolving models, and AI safety.
IT之家 · 70d ago
Resolution (formerly Sequent) has received a $160M grant from Coefficient Giving to accelerate research into high-confidence AI alignment. The funding aims to bridge the resource gap between nonprofit research and frontier labs by scaling compute and research teams.
AI Alignment Forum · 72d ago
ByteDance and Alibaba are discontinuing their AI companion features to align with new Chinese regulations governing human-AI interaction. This move reflects the tightening oversight of generative AI applications in the region.
Bloomberg Technology · 75d ago
UK Foreign Secretary Yvette Cooper warns that AI poses a 'Hiroshima-style' threat to humanity without global regulatory frameworks. She emphasizes that establishing international AI rules will be a central focus of foreign policy over the next two years.
The Guardian Technology · 76d ago
The UK Cabinet Office rejected the idea that dangerous AI could simply be shut down with a kill switch. The government says AI systems cannot simply be turned off, highlighting the limits of emergency-control assumptions.
BBC Technology · 8d ago
Researchers at Anthropic have raised concerns about the accelerating pace of AI development. Their warnings echo broader calls from AI experts for a slowdown in development.
New York Times Technology · 10d ago