Redis array: short story of a long development process

Source: Hacker News

antirez – 40 minutes ago · 3545 views

I started working on the new Array data type for Redis in the first days of January. The PR landed in the repository only now, so this code was cooked for four months. I worked on the implementation part‑time (though many weeks were actually full‑time; detaching from the keyboard can be complicated), and even before LLMs the implementation was something I could have done in four months. What changed is that, in the same time span, I was able to do a lot more. This is the short story of what happened.

Specification (Month 1)

• Wrote a detailed specification document covering:
  • Rationale for the new data type
  • C structures and the sparse representation used
  • Exact semantics of the array cursor for ring buffer and ARINSERT
• Initially drafted the spec by hand, then paired with Opus, and later switched to GPT‑5.3 (and subsequently GPT‑5.x) for design and development.
• The specification evolved through back‑and‑forth feedback, intellectual challenges about the best design, compromises, and avoiding over‑engineering.

Implementation (Month 2)

• Began automatic programming with AI assistance, continuously reviewing the generated code.
• Discovered that the initial level of indirection (directory + slices) was insufficient for commands like ARSET myarray 293842948324 foo without huge allocations.
• Revised the design to a super directory of sliced dense directories, each pointing to array slices (default 4096 elements per slice).
  • This kept the “actually an array” representation, met memory‑characteristic goals, and allowed ARSCAN and ARPOP to run in time proportional to the number of existing elements rather than the range span.

Review & Optimization (Month 3)

• Performed a line‑by‑line code review; AI‑generated tests covered most cases, but superficial correctness didn’t guarantee optimality.
• Identified small inefficiencies and design errors, then rewrote many modules manually with AI assistance.
• Conducted extensive stress testing across varied scenarios, gaining confidence in the robustness and usefulness of the implementation.

Extending the Idea: ARGREP and Regular Expressions

• While modeling use cases, I stored markdown files in Redis arrays, realizing they made an excellent match for a centralized knowledge base.
• This led to the creation of ARGREP, which required regular‑expression support.
• Chose the TRE library (thanks to Ville Laurikari) for its safety against pathological patterns in time or space.
• TRE performed poorly on patterns like foo|bar|zap; with GPT assistance I optimized the library, fixed potential security issues, and expanded the test suite.

Reflections

• High‑quality system programming still demands full involvement, but AI provided a safety net for:
  1. Massive, tedious tasks (e.g., adding and testing 32‑bit support later).
  2. A virtual workforce to catch obvious bugs in complex algorithms.
• Writing the extensive initial specification was crucial; it enabled thorough line‑by‑line reviews of sparsearray.c and t_array.c.

Conclusion

I didn’t repeat the detailed use cases here—they’re documented in the PR message:
Redis PR #15162

I believe it’s time for Redis to have a data type where the numerical index is part of the semantics. I hope the Array PR will be accepted soon and that the community can benefit from the new use cases it enables. Feedback is welcome. Thank you.