[Paper] PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases

Source: arXiv - 2511.19949v1

Overview

The paper introduces PolarStore, a cloud‑native storage layer that slashes database storage costs without sacrificing performance. By marrying a purpose‑built hardware compressor (PolarCSD) with a lightweight software tier, the system delivers high compression ratios for massive workloads while keeping the I/O path fast enough for production‑grade relational databases.

Key Contributions

• Dual‑layer compression architecture – combines in‑storage hardware compression with a thin software compressor to capture the best of both worlds.
• Database‑aware I/O optimizations – redesigns critical read/write paths to avoid the latency penalties typical of compression.
• Hardware refinements for stability – adds host‑level safeguards to PolarCSD chips, proven in deployments across thousands of servers.
• Compression‑aware cluster scheduling – a scheduler that places data blocks based on their compressibility, boosting overall space efficiency.
• Real‑world validation – deployed on PolarDB’s production fleet (≈ 100 PB of data), achieving a 3.55× compression ratio and ~60 % cost reduction with performance on par with uncompressed clusters.

Methodology

1. Two‑stage compression pipeline

   • Stage 1 (hardware): Data is streamed to PolarCSD ASICs embedded in the storage nodes. These chips perform fast, lossless compression using a custom dictionary tuned for relational data patterns.
   • Stage 2 (software): A lightweight CPU‑based compressor runs just before data hits the network stack, handling edge cases (e.g., small rows, schema‑specific encodings) that the hardware cannot efficiently process.

2. I/O path redesign

   • Reads bypass the software stage when the requested block is already in a “hot” compressed form, allowing direct decompression on the storage node.
   • Writes batch small rows together to improve hardware compressor utilization, reducing per‑row overhead.

3. Stability enhancements

   • Added watchdog logic and error‑correction buffers inside PolarCSD to prevent host crashes caused by rare hardware glitches.

4. Compression‑aware scheduler

   • The scheduler monitors per‑table compressibility metrics and co‑locates highly compressible tables on the same physical disks, maximizing the effective storage density of each node.

5. Evaluation

   • Benchmarks were run on a production‑scale PolarDB cluster (thousands of nodes, 100 PB total).
   • Workloads included OLTP (TPC‑C), OLAP (TPC‑DS), and mixed‑mode queries typical of SaaS applications.

Results & Findings

• Space efficiency: The dual‑layer approach yields a 3.55× reduction in raw storage usage, directly translating to lower SSD/HDD provisioning and cloud storage fees.
• Performance impact: Latency and throughput penalties are within single‑digit percentages, essentially invisible to most applications.
• Scalability: The system scales linearly; adding more PolarCSD‑enabled nodes preserves both compression ratio and performance characteristics.

Practical Implications

• Cost‑effective scaling: Cloud‑native DBaaS providers can offer larger database footprints to customers without proportionally increasing storage spend.
• Simplified ops: Because compression is transparent to the database engine, developers do not need to rewrite queries or manage separate compression layers.
• Hardware‑accelerated workloads: The PolarCSD ASIC can be integrated into existing storage servers (NVMe‑over‑Fabric, disaggregated storage) with minimal firmware changes, making retrofits feasible.
• Better multi‑tenant isolation: Higher compression ratios free up space for additional tenants on the same physical cluster, improving overall utilization.
• Potential for edge‑cloud: The lightweight software stage means the same architecture can be deployed on smaller edge nodes where full‑blown hardware compressors are impractical.

Limitations & Future Work

• Workload dependency: Compression gains vary with data entropy; highly random or already compressed blobs (e.g., media files) see modest benefits.
• Hardware availability: PolarCSD is a custom ASIC; broader adoption hinges on open‑sourcing the design or offering it as a commodity accelerator.
• Dynamic re‑compression: The current system does not automatically recompress data when its access pattern changes; future work could add adaptive re‑compression triggers.
• Security considerations: While the compression is lossless, integrating encryption (e.g., TLS‑offload) with the hardware stage remains an open challenge.

PolarStore demonstrates that thoughtful co‑design of hardware and software can break the traditional performance‑vs‑compression trade‑off, delivering real cost savings for cloud‑native databases without compromising the developer experience.

Authors

• Qingda Hu
• Xinjun Yang
• Feifei Li
• Junru Li
• Ya Lin
• Yuqi Zhou
• Yicong Zhu
• Junwei Zhang
• Rongbiao Xie
• Ling Zhou
• Bin Wu
• Wenchao Zhou

Paper Information

• arXiv ID: 2511.19949v1
• Categories: cs.DC, cs.DB
• Published: November 25, 2025
• PDF: Download PDF