๐Ÿ•ท๏ธ
SetupAI
๐Ÿฆ™Stalecollected in 67m

Cohere 30B A3B model GGUF release by Unsloth

Cohere 30B A3B model GGUF release by Unsloth
PostLinkedIn
๐Ÿฆ™Read original on Reddit r/LocalLLaMA
#gguf#local-inference#quantizationcohere-30b-a3bcohereunslothllama.cpp

๐Ÿ’กNew 30B model from Cohere is now available in GGUF format for local inference via Unsloth.

โšก 30-Second TL;DR

What Changed

GGUF format release for Cohere 30B A3B

Why It Matters

The availability of GGUF formats allows developers to run high-performance models on consumer hardware, democratizing access to large-scale AI.

What To Do Next

Download the GGUF file from Hugging Face and test it using llama.cpp to evaluate its performance on your local hardware.

Who should care:Developers & AI Engineers

Key Points

  • โ€ขGGUF format release for Cohere 30B A3B
  • โ€ขEnables local execution of the new 30B model
  • โ€ขPotentially linked to llama.cpp PR #24260 for architecture support

๐Ÿง  Deep Insight

Web-grounded analysis with 27 cited sources.

๐Ÿ”‘ Enhanced Key Takeaways

  • โ€ขThe Cohere 30B A3B model is officially named "North Mini Code 1.0" and represents Cohere's inaugural agentic coding model, specifically designed for tasks such as code generation, agentic software engineering, and terminal interactions.
  • โ€ขThis model employs a Mixture-of-Experts (MoE) architecture, featuring 30 billion total parameters but activating only 3 billion parameters at any given time, which significantly reduces inference costs and enables efficient local deployment on consumer-grade hardware.
  • โ€ขCohere released North Mini Code 1.0 under the permissive Apache 2.0 license, marking a strategic shift towards open-source accessibility for its models, a departure from previous releases like Command R and R+ which utilized more restrictive licenses.
  • โ€ขUnsloth, known for its optimized GGUF releases, also provides Unsloth Studio, a platform that facilitates running and fine-tuning various GGUF and Safetensors models locally, offering features such as tool-calling and web search capabilities.
  • โ€ขThe GGUF (GPT-Generated Unified Format) format, developed by the llama.cpp project, is specifically optimized for CPU efficiency and integrates quantization, making large language models feasible on consumer-grade hardware by bundling model parameters and metadata into a single, self-contained file.
๐Ÿ“Š Competitor Analysisโ–ธ Show
Feature/MetricCohere North Mini Code 1.0 (30B A3B)Mistral Devstral Small 2 (24B Dense)Qwen3.6 35B A3BGitHub Copilot / Cursor / Claude Fable 5
ArchitectureSparse Mixture-of-Experts (MoE), 30B total / 3B active parametersDense model, 24B parametersSparse Mixture-of-Experts (MoE), 35B total / 3B active parametersProprietary / Managed Service
LicenseApache 2.0(Not specified in search, likely more restrictive than Apache 2.0 for enterprise use)Apache 2.0Proprietary / Commercial (per-usage or subscription)
DeploymentLocal (GGUF, llama.cpp, Unsloth Studio), Hugging Face, API(Likely API and potentially local with GGUF conversions)Local (GGUF, llama.cpp, Unsloth Studio), Hugging FaceCloud-based API / Managed Service (no on-premises option for Copilot/Cursor/Fable 5)
Output Throughput2.8x higher than Devstral Small 2 (internal tests), ~199 tokens/second (Artificial Analysis)(Baseline for comparison)~30 tokens/second (on 6GB VRAM with llama.cpp)Claude Fable 5: $50 per million output tokens
Inter-token Latency30% decrease over Devstral Small 2 (internal tests), 0.25 seconds (Artificial Analysis)(Baseline for comparison)(Not specified for direct comparison)(Not specified for direct comparison)
Intelligence Index (Artificial Analysis)27.6(Not specified, but Mistral Small 4 is 27.8)(Not specified for this exact model, but Qwen3.6 35B A3B is 35.2 on Coding Index)(Not specified for direct comparison)
Coding Index (Artificial Analysis)33.4(Not specified)35.2(Not specified for direct comparison)
Context Window256K input, 64K output(Not specified)256K natively, extendable up to 1M with Yarn(Not specified, but generally large for cloud models)
Target Use CaseAgentic coding, software engineering, terminal tasks(Coding-focused)Agentic Coding, Agentic Browser-Use, foundational coding tasksCoding assistance, code generation

๐Ÿ› ๏ธ Technical Deep Dive

  • Model Name: North Mini Code 1.0
  • Architecture: Decoder-only Transformer-based sparse Mixture-of-Experts (MoE) model.
  • Parameters: 30 billion total parameters, with 3 billion active parameters during inference.
  • Experts: Comprises 128 experts, with 8 experts activated per token.
  • Layers: Features 49 layers.
  • Hidden Size: Utilizes a hidden size of 2048.
  • Attention Mechanism: Employs Grouped Query Attention (GQA) with 32 queries and 4 KV heads. It uses a hybrid attention mechanism, interleaved between sliding-window attention with RoPE (Rotary Positional Embeddings) and global attention without positional embeddings, in a 3:1 ratio.
  • Context Length: Supports a 256K input context length and a 64K output length.
  • Training: Trained through two stages of supervised fine-tuning, followed by reinforcement learning with verifiable rewards across over 70,000 tasks spanning approximately 5,000 repositories.
  • GGUF Format: The GGUF (GPT-Generated Unified Format) is a binary file format optimized for efficient local LLM inference, designed for CPU efficiency, and integrates quantization to reduce model size and computational needs. It bundles model weights, architecture metadata, tokenizer, configuration, and quantization information into a single, self-contained file.

๐Ÿ”ฎ Future ImplicationsAI analysis grounded in cited sources

The Apache 2.0 licensing of Cohere's North Mini Code 1.0 will accelerate its adoption in enterprise and sovereign AI initiatives.
The permissive Apache 2.0 license allows commercial use and modification without restrictive clauses, which is crucial for enterprises and governments prioritizing data privacy and control, fostering broader integration into secure, on-premises environments.
The efficient MoE architecture of North Mini Code 1.0 will drive further innovation in developing powerful, locally deployable agentic AI models.
By achieving strong performance with a small active parameter footprint, this model demonstrates that advanced agentic capabilities can be run on consumer-grade hardware, lowering the barrier to entry for developers and encouraging the creation of more specialized, efficient local AI agents.
Unsloth's role in quickly releasing GGUF versions of new models will solidify its position as a key enabler for local LLM inference and fine-tuning.
Unsloth's consistent and optimized GGUF releases, coupled with tools like Unsloth Studio, directly address the demand for accessible local execution of cutting-edge models, making them readily available to a wider developer community.

โณ Timeline

2017
Aidan Gomez, a co-founder of Cohere, co-authors the seminal "Attention Is All You Need" Transformer paper.
2019
Cohere is founded in Toronto by Aidan Gomez, Nick Frosst, and Ivan Zhang, focusing on enterprise-grade natural language processing.
2021
Cohere secures Series A funding and launches its first API, marking its entry into commercial AI services.
2024
Cohere releases Command R and Command R+ models, specifically optimized for retrieval-augmented generation (RAG) at enterprise scale.
2026-05-20
Cohere unveils Command A+, a 218-billion-parameter Sparse Mixture-of-Experts (MoE) model, released under the Apache 2.0 license.
2026-06-09
Cohere releases North Mini Code 1.0 (30B A3B), its first agentic coding model, as an open-source model under the Apache 2.0 license.
๐Ÿฆ™Read original article on Reddit r/LocalLLaMA
๐Ÿ“ฐ

Weekly AI Recap

Read this week's curated digest of top AI events โ†’

๐Ÿ‘‰Related Updates

Same topic

Explore #gguf

Same product

More on cohere-30b-a3b

Same source

Latest from Reddit r/LocalLLaMA

AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ†—