Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText

Source: Google Developers Blog

Continuous Checkpointing in Orbax and MaxText

The newly introduced continuous checkpointing feature in Orbax and MaxText is designed to optimize the balance between reliability and performance during model training. It directly addresses the shortcomings of conventional fixed‑frequency checkpointing.

Why move away from fixed‑frequency checkpointing?

• Fixed intervals can compromise reliability if they are too sparse.
• They can also bottleneck performance when checkpoints are taken too often.

Continuous checkpointing avoids these trade‑offs by adapting to the actual I/O conditions of the training job.

How continuous checkpointing works

• The system maximizes I/O bandwidth by initiating a new save operation asynchronously.
• A new checkpoint is started only after the previous one has completed successfully, eliminating overlap and reducing contention.

Benchmark results

• Benchmarks show a significant reduction in checkpoint intervals.
• The approach leads to substantial resource conservation, which is especially valuable for large‑scale training jobs where the mean‑time‑between‑failure (MTBF) is short.