๐ก๏ธCloudflare BlogโขStalecollected in 80m
Solving hidden lock contention in petabyte-scale ClickHouse clusters
๐กLearn how to debug silent performance bottlenecks in large-scale data infrastructure used for AI training pipelines.
โก 30-Second TL;DR
What Changed
Identified severe lock contention in ClickHouse's query planner during partitioning operations.
Why It Matters
This highlights the importance of deep observability in large-scale data infrastructure. It serves as a reminder that standard metrics may miss critical performance bottlenecks in complex distributed systems.
What To Do Next
If you run high-scale ClickHouse clusters, audit your query planner logs and partition change frequency to proactively identify potential lock contention.
Who should care:Developers & AI Engineers
Key Points
- โขIdentified severe lock contention in ClickHouse's query planner during partitioning operations.
- โขStandard metrics failed to surface the bottleneck, requiring deep-dive debugging.
- โขContributed upstream patches to ClickHouse to improve concurrency and performance.
- โขMaintained petabyte-scale data integrity while resolving critical billing job stalls.
๐ง Deep Insight
Web-grounded analysis with 22 cited sources.
๐ Enhanced Key Takeaways
- โขThe identified bottleneck was specifically an exclusive lock in ClickHouse's query planner, which was being acquired for a read-only operation (reading the parts list) during partitioning changes, leading to contention.
- โขCloudflare's internal 'Ready-Analytics' system, established in early 2022, allows hundreds of internal teams to stream data into a single, massive ClickHouse table. This system uses a primary key incorporating
namespace,indexID, andtimestampto enable optimal data sorting tailored to each team's query patterns. - โขThe resolution involved Cloudflare contributing upstream patches to ClickHouse, specifically modifying the query planner to use a shared lock instead of an exclusive lock when merely reading the parts list, thereby improving concurrency and preventing stalls.
- โขCloudflare has a history of significant contributions to the ClickHouse open-source project, dating back to at least 2018, including enhancements like aggregate functions (e.g.,
topK,sumMap), IP prefix dictionaries, and optimizations for theSummingMergeTreeengine. - โขClickHouse partitioning, while a crucial data management technique for large tables, can paradoxically degrade query performance if too many partitions are created, leading to excessive merge overhead and increased pressure on internal systems like ClickHouse Keeper in replicated environments.
๐ Competitor Analysisโธ Show
| Feature/Product | ClickHouse | Google BigQuery | Snowflake | Apache Druid | StarRocks |
|---|---|---|---|---|---|
| Primary Use Case | Real-time OLAP, high-throughput analytics, log/event data | Serverless data warehousing, ad-hoc queries, deep GCP integration | Cloud data warehousing, separation of compute/storage, multi-cloud | Real-time OLAP, time-series data, real-time ingestion | Real-time OLAP, multi-table joins, high concurrency |
| Operational Overhead | Manageable, but OSS requires manual tuning; ClickHouse Cloud offers managed experience | Zero operational overhead, fully serverless | Minimal tuning, easy to use | More complex to operate than ClickHouse | Designed for modern enterprises, aims for high performance |
| Performance (Real-time) | Excels in real-time analytics, sub-second query latency | Slower than ClickHouse for high-frequency queries, batch-oriented | Not designed for real-time analytics | Optimized for sub-second query latency on streaming data | Blazing-fast execution engine, good for complex, high-concurrency queries |
| Pricing Model | Cost-effective at scale compared to cloud warehouses | Per-query pricing, can be expensive at scale | Expensive at scale | Open-source (self-managed), various managed services exist | Open-source (self-managed), commercial offerings available |
| Scalability | Linearly scalable, supports sharding and replication for petabyte scale | Truly serverless, petabyte-scale analytics without infrastructure management | True separation of compute and storage, scales easily | Extreme scale with strict latency | Good single-server performance, but scaling clusters can be challenging (manual re-balancing) |
| Key Differentiators | Columnar storage, vectorized execution, open-source community, strong for log/event data | Serverless, deep ecosystem integration, ideal for ad-hoc analysis | Multi-cloud, time travel, zero-copy cloning, enterprise features | Optimized for time-series, real-time ingestion, complex aggregates | Strong multi-table join performance, cost-based optimizer, materialized views |
๐ ๏ธ Technical Deep Dive
- ClickHouse Partitioning Mechanism: Data is organized into logical segments called 'parts' based on a specified key using the
PARTITION BYclause during table creation. Each unique value of the partitioning expression forms its own physical partition on disk. This mechanism is primarily for data management, enabling efficient operations likeDROP PARTITIONandATTACH PARTITION, and can improve query performance by allowing ClickHouse to skip irrelevant partitions (partition pruning). - Lock Contention Specifics: The issue stemmed from the ClickHouse query planner acquiring an exclusive lock on the parts list during partitioning changes. This lock, intended for write operations, was being used for a read-only task, blocking concurrent queries and leading to severe latency.
- Cloudflare's Solution: The fix involved modifying the ClickHouse codebase to replace the exclusive lock with a shared lock for the query planner's read access to the parts list. This change allows multiple read operations to proceed concurrently, resolving the contention.
- ClickHouse Architecture for Scale: ClickHouse leverages columnar storage, aggressive data compression, and vectorized query execution. It uses sparse indexes, where the primary key defines the sort order within data parts, enabling fast identification of relevant data. For petabyte-scale deployments, it supports sharding and replication across multiple nodes.
- Cloudflare's Ready-Analytics System: This system uses a standard schema with a primary key of
(namespace, indexID, timestamp). TheindexIDis a string field that allows each of the hundreds of internal teams to define a sorting order optimal for their specific queries, all within a single, massive ClickHouse table.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
ClickHouse will see improved stability and performance for large-scale, dynamic partitioning operations.
Cloudflare's upstream patches directly address a hidden bottleneck in the query planner, preventing similar issues for other users and improving overall system robustness, especially in environments with frequent partitioning changes.
ClickHouse development will continue to prioritize concurrency and distributed indexing for petabyte-scale workloads.
Recent ClickHouse features like 'Index Sharding' (April 2026), which distributes index analysis across replicas, alongside Cloudflare's contributions, indicate an ongoing focus on optimizing performance and concurrency at extreme scales.
Enterprise adoption of ClickHouse for critical billing and analytics systems will be further reinforced.
Cloudflare's successful identification and resolution of a critical issue in its billing pipeline, coupled with its ongoing contributions, demonstrates ClickHouse's adaptability and the strength of its open-source community for demanding enterprise workloads.
โณ Timeline
2009
Alexey Milovidov and team begin experimental project for real-time analytical reports, which evolves into ClickHouse.
2012
ClickHouse launches internally to power Yandex.Metrica, the second-largest web analytics platform.
2016
ClickHouse is open-sourced under the Apache 2 license; Cloudflare begins running ClickHouse in production.
2018
Cloudflare engineers contribute various patches and features upstream to ClickHouse, including aggregate functions and Kafka table engine optimizations.
2021
ClickHouse, Inc. incorporates as a separate entity and receives initial $50M Series A investment.
2022
Cloudflare builds its 'Ready-Analytics' system, a massive ClickHouse table for hundreds of internal teams.
2026-05-14
Cloudflare publishes an article detailing the identification and resolution of severe lock contention in ClickHouse's query planner during partitioning changes.
๐ Sources (22)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
- Google Search Source
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Cloudflare Blog โ