๐ฅPyTorch BlogโขStalecollected in 49m
Run PyTorch Models on Apple Silicon via MLX Delegate
๐กUnlock native GPU acceleration for your PyTorch models on Apple Silicon with the new MLX delegate.
โก 30-Second TL;DR
What Changed
Enables GPU-accelerated inference for PyTorch models on Apple Silicon Macs.
Why It Matters
This update significantly lowers the barrier for developers to deploy high-performance AI models locally on Mac hardware. It allows for more efficient use of Apple's unified memory architecture for machine learning tasks.
What To Do Next
If you are building on-device AI for macOS, integrate the ExecuTorch MLX delegate into your pipeline to benchmark inference speed improvements.
Who should care:Developers & AI Engineers
Key Points
- โขEnables GPU-accelerated inference for PyTorch models on Apple Silicon Macs.
- โขIntegrates directly with Apple's MLX framework for optimized performance.
- โขPart of the ExecuTorch ecosystem for edge and on-device AI deployment.
๐ง Deep Insight
Web-grounded analysis with 25 cited sources.
๐ Enhanced Key Takeaways
- โขThe ExecuTorch MLX delegate specifically integrates quantized GEMM kernels from Apple's MLX framework, enabling efficient int4 inference for PyTorch models on Apple Silicon, which is crucial for memory-constrained edge devices.
- โขApple's MLX framework, which the delegate leverages, is an array framework designed for Apple hardware, featuring a NumPy-like API, unified memory optimization, lazy computation, and dynamic graph construction, making it suitable for both research and deployment.
- โขExecuTorch, the broader ecosystem this delegate belongs to, is an end-to-end solution for on-device AI inference across diverse edge devices, offering superior portability and a significantly smaller memory footprint compared to its predecessor, PyTorch Mobile, by utilizing the PyTorch 2 compiler and export functionality.
- โขPrior to the MLX delegate, PyTorch already supported GPU acceleration on Apple Silicon via the Metal Performance Shaders (MPS) backend, introduced in May 2022, which optimizes compute performance using kernels fine-tuned for Apple's Metal GPU families.
๐ Competitor Analysisโธ Show
Comparison of PyTorch on Apple Silicon Delegates/Frameworks
| Feature | ExecuTorch MLX Delegate | PyTorch MPS Backend | Core ML (as a deployment target) |
|---|---|---|---|
| Purpose | Optimized PyTorch inference on Apple Silicon via MLX. | GPU-accelerated PyTorch training/inference on Apple Silicon via Metal Performance Shaders. | On-device model deployment for Apple native applications. |
| Underlying Tech | Apple's MLX framework, specifically leveraging its optimized kernels (e.g., quantized GEMM for int4 inference). | Apple's Metal Performance Shaders (MPS) framework, mapping ML graphs to MPS Graph and tuned kernels. | Leverages Apple Neural Engine (ANE), GPU (Metal), and CPU, with automatic hardware selection. |
| PyTorch Integration | Part of the ExecuTorch ecosystem, uses torch.export for model capture and compilation. | Native PyTorch backend, accessible via torch.device("mps"). | Requires model conversion from PyTorch to .mlmodel format for deployment. |
| Optimization Focus | Inference, particularly for memory-efficient quantized models (e.g., int4). | General GPU acceleration for both training and inference. | Broad hardware utilization (ANE, GPU, CPU) for low-power, efficient inference. |
| Memory Model | Leverages MLX's unified memory architecture for efficient data handling without transfers. | Leverages Apple Silicon's unified memory architecture, allowing GPU direct access to full memory. | Utilizes unified memory, but can experience runtime allocation issues due to weight decompression and padding. |
| Quantization | Specifically ports int4 quantized GEMM kernels from MLX. | Supports 16-bit precision, though full torch.compile support was still developing as of mid-2023. | Supports INT8 and INT4, with INT4 optimized for GPU rather than ANE. |
| Performance Notes | MLX inference on M2 Max showed significant narrowing of performance gap compared to CUDA GPUs for transformer models. | M1 Pro GPU can be 8.8x faster than M1 Pro CPU for training, but approximately 13.77x slower than an Nvidia A6000. | INT8 can be 3.3x faster than INT4 on the Neural Engine; MLX and CoreML can run concurrently on separate hardware. |
๐ ๏ธ Technical Deep Dive
- MLX Framework Core: MLX is an array framework by Apple Machine Learning Research, offering a Python API similar to NumPy, along with C++, C, and Swift bindings. It includes higher-level packages like
mlx.nnandmlx.optimizersfor neural network construction. - Unified Memory Architecture: MLX is specifically designed to leverage Apple Silicon's unified memory, allowing arrays to reside in shared memory and operations to be performed across CPU and GPU without data transfers, optimizing memory usage.
- Lazy Computation and Dynamic Graphs: Computations in MLX are lazy, meaning arrays are only materialized when needed. It uses dynamic graph construction, which avoids slow compilations when argument shapes change and simplifies debugging.
- ExecuTorch Workflow: The process for running PyTorch models with ExecuTorch involves three main steps: 1) Exporting the PyTorch program as a graph using
torch.export, which preserves semantics. 2) Compiling the exported model into an ExecuTorch program, allowing for optimizations like quantization and subgraph compilation. 3) Running the compiled program using a lightweight ExecuTorch runtime. - MLX Delegate Specifics: The ExecuTorch MLX delegate integrates by porting quantized GEMM (General Matrix Multiply) kernels from MLX, specifically for int4 inference. This targets highly efficient, low-precision inference on Apple Silicon GPUs.
- Backend Delegation and Quantization: ExecuTorch supports selective backend delegation, enabling parts of a model to run on specialized accelerators like the Apple Neural Engine, with CPU fallback. It also features robust memory planning and quantization capabilities, building on TorchAO, to reduce model size and inference latency.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Apple Silicon will see increased adoption for local AI development and deployment.
The ExecuTorch MLX delegate provides a highly optimized and streamlined path for PyTorch models, making Apple Silicon more competitive for on-device AI workloads and reducing reliance on cloud resources or specialized hardware.
Privacy-preserving and real-time AI applications on Apple devices will become more sophisticated.
ExecuTorch's focus on efficient on-device inference, combined with MLX's hardware-specific optimizations, enables more complex AI models to run locally, which is critical for applications requiring data privacy and low latency.
Further integration and convergence between PyTorch's edge ecosystem and Apple's native ML frameworks are likely.
The introduction of the MLX delegate signifies a deeper technical collaboration between PyTorch and Apple, suggesting future efforts to create more seamless development and deployment workflows across their respective AI ecosystems.
โณ Timeline
2016
PyTorch project initiated as a successor to Torch.
2018-12
PyTorch 1.0 released, combining research-friendly design with production capabilities.
2020
Apple announced its transition to ARM-based Apple Silicon processors.
2022-05-18
PyTorch announced GPU-accelerated training support on Mac via the Metal Performance Shaders (MPS) backend.
2023-11
Apple introduced the MLX framework for efficient machine learning on Apple Silicon.
2025-10-22
ExecuTorch 1.0 released, promoting Apple Core ML backend to production-ready status and expanding hardware support.
2026-05-18
PyTorch announced the ExecuTorch MLX delegate for optimized GPU-accelerated inference on Apple Silicon.
๐ Sources (25)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: PyTorch Blog โ