MoE Refusals Routed via Experts in Abliteration

🔑 Enhanced Key Takeaways

• •The 'Abliteration' technique leverages the Gram-Schmidt process to identify and neutralize specific activation vectors, effectively decoupling model refusal behaviors from core reasoning capabilities.
• •The Qwen3.5-397B-A17B architecture utilizes a specialized gating mechanism that prioritizes expert routing based on input semantic clusters, which explains why simple weight-baking is insufficient for complete safety-layer removal.
• •The research highlights a critical vulnerability in large-scale MoE models where safety-aligned experts are hard-coded into the routing logic, necessitating 'inference hooks' to intercept and redirect activations at the router level.

🛠️ Technical Deep Dive

• •Model Architecture: Qwen3.5-397B-A17B (Mixture-of-Experts) with 17B active parameters per token.
• •Abliteration Method: Orthogonal projection of activation vectors onto a subspace defined by refusal-inducing prompts, utilizing Gram-Schmidt orthonormalization to minimize impact on non-refusal tokens.
• •Inference Hook Mechanism: A custom runtime layer that intercepts the router's top-k expert selection, forcing a bypass of 'safety-aligned' experts when specific refusal-triggering semantic patterns are detected.
• •Hardware Constraints: Execution on Mac Studio M3 Ultra (Unified Memory Architecture) requires aggressive quantization (likely 4-bit or EXL2) to fit the 397B parameter footprint within available VRAM.

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline