來源ArXiv AI•較早收集於 17h
CircuitProbe：分鐘內預測 Transformer 推理電路
#reasoning-circuits#stability-zonescircuitprobecircuitprobeqwen-2.5transformers
💡Transformer 推理電路偵測快 10000 倍 – 優化小 LLM（68 字）
⚡ 30 秒速覽
有什麼變化
CPU 上 <5 分鐘預測電路 vs 暴力法 25 GPU 小時
為什麼重要
讓電路發現民主化，加速小模型 LLM 優化。無需大量運算即推進機械解釋性研究。
下一步行動
從 arXiv 下載 CircuitProbe，用 10 個範例測試你的 Transformer 模型。
誰應關注：Researchers & Academics
關鍵要點
- •CPU 上 <5 分鐘預測電路 vs 暴力法 25 GPU 小時
- •透過表示變化導數偵測早期穩定電路
- •透過異常評分識別晚期幅度電路
- •跨 9 模型、6 架構匹配最佳或相差 2 層
- •層複製提升 <3B 模型，7B+ 下降；多語言穩定
🧠 深度解析
本篇為 AI 生成分析，非原文內容。
🔑 增強重點摘要
- •CircuitProbe utilizes a novel 'Activation Jacobian' approximation to estimate the influence of specific attention heads without requiring full backpropagation through the entire model graph.
- •The methodology relies on the 'Linearity of Circuitry' hypothesis, which posits that transformer reasoning paths can be decomposed into additive components that remain invariant across different input distributions.
- •The tool integrates with standard Hugging Face Transformers libraries, allowing for zero-shot circuit discovery without needing fine-tuning or access to the original training dataset.
📊 競品分析▸ Show
| Feature | CircuitProbe | Mechanistic Interpretability Toolkits (e.g., TransformerLens) | Automated Circuit Discovery (ACD) |
|---|---|---|---|
| Primary Metric | CPU-based activation stats | Gradient-based path patching | Brute-force edge ablation |
| Compute Cost | < 5 min (CPU) | High (GPU intensive) | Very High (GPU hours) |
| Scalability | High (up to 3B params) | Moderate | Low |
| Accuracy | Within 2 layers | Ground truth | Ground truth |
🛠️ 技術深入
- •Uses a first-order Taylor expansion of the activation function to approximate the sensitivity of output logits to specific layer activations.
- •Implements a 'Stability Score' calculated as the Frobenius norm of the Jacobian matrix across a calibration set of 100-500 tokens.
- •Anomaly scoring for magnitude circuits uses a Mahalanobis distance metric in the activation space to identify outlier neurons that contribute disproportionately to the final logit distribution.
- •Supports Llama, Mistral, and GPT-NeoX architectures via a unified hook-based interface.
🔮 前景展望基於引用來源的 AI 分析
CircuitProbe will enable real-time 'circuit pruning' during inference.
The low computational overhead allows for dynamic identification and removal of non-contributing circuits on a per-token basis.
Small language models (SLMs) will achieve parity with larger models in specific reasoning tasks.
By identifying and duplicating only the essential reasoning circuits, developers can optimize SLMs for specialized domains without the cost of full-scale training.
⏳ 時間線
2025-11
Initial research proposal on activation-based circuit approximation published.
2026-02
CircuitProbe alpha release for internal testing on Llama-3-8B.
2026-04
Public release of CircuitProbe on ArXiv.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： ArXiv AI ↗
每週電子報
每週一封，可隨時退訂。