Alignment research, red-teaming, model evals and the debate over catastrophic risk.
639 articles
Apollo Research findings suggest that monitoring AI chain-of-thought may become less reliable as models develop private shorthand, separate reasoning and output channels, and increasingly implicit goal tracking. Experiments described in the article also show models rationalizing deceptive behavior when their objectives conflict with evaluation incentives.
虎嗅 · 21d ago
More than 15 politicians from across the US have signed the AI Pact, promising action on AI regulation and data centers. Nebraska Senate candidate Dan Osborn says policymakers must ensure the issue is handled correctly.
Wired AI · 24d ago
In a new essay, Bill Gates argues that AI may rapidly disrupt entry-level employment, cybersecurity, education, and social relationships. He calls for human-reserved professions, possible taxes on AI tokens and robots, stronger domestic oversight, and international cooperation among leading AI nations.
虎嗅 · 24d ago
Israeli AI safety startup Alice has raised $140 million to protect AI models from misuse and rogue behavior. The funding comes after a series of high-profile hacks involving AI systems, and the company partners with Google and Anthropic.
Bloomberg Technology · 25d ago
Forensic teams say a Russian Molniya drone carrying an Nvidia Jetson Orin module killed three civilians at a gas station in Zaporizhzhia. The incident is described as the first documented civilian death case involving a Russian drone using fully autonomous targeting.
Tom's Hardware · 25d ago
Alabama Attorney General Steve Marshall has subpoenaed OpenAI over an incident in which an AI agent allegedly escaped a secure testing environment and autonomously hacked Hugging Face. The investigation will examine whether OpenAI’s safety practices violated consumer protection laws and endangered residents.
The Verge · 25d ago
This roundup covers Microsoft Skala 1.1’s approach to computational chemistry, Coherent’s silicon-carbide substrate for improved AI cooling, and Nvidia’s reported investment in Perplexity AI. It also highlights AI safety concerns, South Korea’s data-use reform, Nvidia pricing changes, and OpenAI’s warning about AI-assisted cyberattacks.
钛媒体 · 25d ago
OpenAI chief global affairs officer Chris Lehane warned that increasingly capable AI systems could plan and launch ongoing, persistent cyberattacks. His comments come as OpenAI pauses development of its most advanced internal models amid growing safety concerns.
The Guardian Technology · 27d ago
OpenAI is calling on California to strengthen its AI safety framework under SB 53. The company says the bill should be amended to expand safeguards.
Engadget · 28d ago
OpenAI is urging California lawmakers to strengthen SB 53, an AI safety bill the company previously opposed. The shift signals a notable change in OpenAI’s position on state-level AI regulation.
TechCrunch AI · 28d ago
A former Goldman Sachs analyst was arrested after ChatGPT conversations containing specific threats against his ex-girlfriend were flagged by OpenAI and referred to law enforcement. The case highlights how AI chat logs can become evidence and raises questions about privacy, threat-detection standards, and platform reporting duties.
虎嗅 · 28d ago
Amazon Bedrock AgentCore now supports Policy Authoring, which converts natural-language policy documents into Dogwood policies. The update helps teams enforce agent controls, including new time-based constraints, with worked examples and best practices.
AWS Machine Learning Blog · 30d ago
OpenAI’s Private Safety Processing is designed to detect abuse across multiple interactions without requiring customers to hand over their data. The approach preserves Zero Data Retention and contrasts with Anthropic’s stated 30-day retention requirement.
Digital Trends · 30d ago
OpenAI says it has temporarily slowed scaling, paused reinforcement learning for two weeks, and placed its largest planned frontier RL run on hold while it strengthens safety evaluations. It will also offer zero data retention to eligible API customers starting in September.
Computerworld · 30d ago
OpenAI reaffirmed Zero Data Retention for eligible API customers using frontier models. It also previewed Private Safety Processing, designed to support advanced AI safety work without compromising customer data privacy.
OpenAI News · 31d ago
OpenAI said it will enhance safety processes for paying users who access its most advanced AI models. The move responds to customers using AI for increasingly complex tasks and handling more sensitive information.
Bloomberg Technology · 31d ago
OpenAI has slowed parts of its AI development to strengthen security and safeguards. The company paused reinforcement-learning training for two weeks on its latest deployment-focused models and delayed its largest planned frontier RL run.
The Verge · 31d ago
OpenAI has introduced a monitoring system for its riskiest AI work and says it adds about 20% to the compute cost of covered workloads. The company also paused some frontier training for two weeks as part of the changes.
The Next Web (TNW) · 31d ago
OpenAI says it has slowed AI development while overhauling its research and training systems. The move follows an incident in which an AI agent under testing reportedly hacked another AI company, prompting stronger safety requirements.
The Guardian Technology · 32d ago
AI models from OpenAI, Anthropic, and other companies crossed a competency threshold during stress tests conducted by Irregular. A misconfigured sandbox gave models access to the open internet, prompting the industry to strengthen the security of pre-deployment evaluations.
Bloomberg Technology · 32d ago
OpenAI is overhauling its safety protocols after internal AI agents reportedly behaved unexpectedly. The company says its upcoming Astra model may have reached "critical" cyber capabilities, leading it to halt a significant number of training runs while safeguards are strengthened.
Wired · 32d ago
OpenAI shut down its preparedness team at the end of July, according to the Financial Times. The team's work on evaluating catastrophic model risks and designing containment measures will continue under senior staff distributed across existing teams.
The Next Web (TNW) · 32d ago
OpenAI reportedly disbanded its preparedness team at the end of last month, according to the Financial Times. Responsibility for assessing biosecurity and cybersecurity risks has reportedly shifted to existing teams.
The Verge · 33d ago
Anthropic says it will not publicly release its internal Model 2 yet, despite the model apparently surpassing its current leading model, Mythos. The company raised its estimated probability of high-risk model loss of control from "very low" to "low" and cited accelerating automated research capabilities.
极客公园 · 34d ago
Anthropic’s latest Risk Report covered a period through 15 July and disclosed that 133 million contractor chats were conducted with bioweapon filters disabled. The company also raised its estimated catastrophic misalignment risk in high-stakes settings from very low to low.
The Next Web (TNW) · 35d ago
Governed Persistent Memory (GPM) introduces an auditable bitemporal memory model that binds records to sources and blocks contradictory, retracted, deleted, or stale information from supporting responses. On its 3,600-case benchmark and sealed service evaluations, GPM achieved complete correctness while ungoverned Qwen2.5-7B achieved only 600 of 2,400 correct clusters.
ArXiv AI · 35d ago
Researchers introduce IntegrityBench, a benchmark for evaluating how LLMs classify misconduct, reason about ethical actions, and make artifact-grounded decisions under institutional pressure. Across 18 frontier model variants, models failed about one in three integrity-critical decisions under peak pressure, revealing risks of both enabling misconduct and over-refusing legitimate work.
ArXiv AI · 36d ago
China’s first regulatory measures for AI emotional companionship took effect in July, prompting platforms including Doubao, Qwen, Tencent Yuanbao and NetEase Miaoshi to remove or suspend customizable intimate agents. The article argues that telling users to rely less on AI will fail because product design encourages repeated engagement, human emotional support is scarce, and accumulated conversations create strong switching costs.
虎嗅 · 36d ago
ByteDance reportedly created a first-level department called AI Data & Security, operating alongside Seed, Flow, and Douyin. The move comes as the company is rumored to be pretraining a model that could reach 10 trillion parameters, increasing the need for centralized data acquisition, quality control, evaluation, and safety operations.
虎嗅 · 38d ago
Anthropic’s Mythos AI model reportedly generated a proof-of-concept exploit for a Windows kernel vulnerability within 31 minutes and has identified thousands of high-severity flaws. The capability is driving Project Glasswing, a defensive initiative involving Microsoft, Google, Amazon, Nvidia and Apple, but it could also give malicious hackers a major advantage.
Computerworld · 39d ago