🤝Together AI Blog•Mar 10, 2026Stalecollected in 34h

Together GPU Clusters Add Autoscaling & Self-Healing

Post LinkedIn

🤝Read original on Together AI Blog

#autoscaling#observability#self-healing#rbactogether-gpu-clusterstogether-ai

💡Production GPU clusters with autoscaling & self-healing cut AI infra costs 30-50%.

⚡ 30-Second TL;DR

What Changed

Built-in autoscaling for dynamic resource scaling

Why It Matters

This reduces ops overhead for AI teams, enabling focus on development over maintenance. It improves cost-efficiency and reliability for large-scale ML training and inference.

What To Do Next

Who should care:Enterprise & Security Teams

Key Points

•Built-in autoscaling for dynamic resource scaling
•RBAC for secure role-based access control
•Full-stack observability for comprehensive monitoring
•Self-healing node repair for automatic failure recovery

🧠 Deep Insight

Background and context from public sources — not the original article. 8 sources cited.

🔑 Enhanced Key Takeaways

•Together AI is reportedly pursuing a $1 billion funding round amid these infrastructure upgrades.[1]
•Autoscaling is powered by Kubernetes Cluster Autoscaler, which monitors GPU-constrained workloads and automatically provisions or decommissions nodes based on real-time demand.[1][2]
•Self-healing involves a three-click process where the control plane cords, drains, and recreates failed nodes, with automatic acceptance tests ensuring clusters pass before marking as ready.[2]
•GPU Clusters support NVIDIA H100, H200, B200, and GB200 GPUs with InfiniBand networking, scaling from 8 to over 4,000 GPUs for distributed training and inference.[3]

🛠️ Technical Deep Dive

•Autoscaling uses Kubernetes Cluster Autoscaler for dynamic node provisioning/deprovisioning based on GPU demand, targeting variable inference and bursty training jobs.[1][2]
•Observability features a dedicated Grafana instance with dashboards for DCGM GPU metrics, InfiniBand/NIC networking telemetry, storage I/O, and Kubernetes health.[2]
•Self-healing node repair: Cordon and drain failed nodes, recreate on new/existing hosts, run acceptance tests (detailed in documentation) before readiness.[2]
•Orchestration options include managed Kubernetes (Kubeadm-based, HA control plane, node autoscaling, Cert Manager) and Slurm on Kubernetes for precise gang scheduling and SSH access.[3]
•Performance enhanced by Together Kernel Collection (including ThunderKittens), achieving 90% faster training on B200 (15,264 tokens/s/GPU) vs H100 (8,080 tokens/s/GPU) for 70B Llama model.[3]

🔮 Future ImplicationsAI analysis grounded in cited sources

Together GPU Clusters will capture more enterprise AI workloads from static provisioning providers.

Dynamic autoscaling and self-healing address key pain points of over/underprovisioning and node failures, enabling efficient scaling for variable training and inference.[1][2]

$1B funding pursuit will accelerate Together AI's GPU infrastructure expansion.

The upgrades coincide with reported funding talks, providing production-ready features to attract enterprise customers and investors in the competitive GPU-as-a-service market.[1][7]

⏳ Timeline

2025-03

Introduced Instant GPU Clusters at NVIDIA GTC 2025.

2025-09

Launched self-service GPU infrastructure with Together Instant Clusters.

2026-03

Announced autoscaling, RBAC, observability, and self-healing for GPU Clusters.

📎 Sources (8)

Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.

🤝Read original article on Together AI Blog

📰

Weekly AI Recap

Read this week's curated digest of top AI events →

👉Related Updates

Same topic

Explore #autoscaling

Same product