The BEHILOS Benchmark: Performance Analysis of Low-Footprint CLI Search Tools
Source: Dev.to
The BEHILOS Benchmark
In his book Unix: A History and a Memoir, Brian Kernighan recounts his favorite grep story from the early days of Unix. Someone at Bell Labs asked whether it was possible to find English words composed only of the letters formed by an upside‑down calculator. The digits on a turned calculator display, 5071438, map to the letter set BEHILOS.
Kernighan grepped the regular expression ^[behilos]*$ against the Webster’s Second International Dictionary (which contained 234 936 words) and found 263 matches – including words he had never seen before.
The current Webster’s Second International Dictionary contains 236 007 words. To reproduce the results, run:
grep ^[behilos]*$ /usr/share/dict/web2This yields 264 matches. The longest words are nine characters long: blissless and booboisie.
What started as a historical footnote quickly made me wonder: how would the old BEHILOS
grepperform as a quick‑and‑dirty benchmark for today’s search tools?
Tools Tested
| Tool | Description |
|---|---|
| grep | Classic Unix tool; historical baseline for text searching. |
| rg (ripgrep) | Modern, extremely fast recursive searcher optimized for large codebases. |
| gawk | GNU AWK implementation; feature‑rich and widely used for text processing. |
| mawk | Lightweight, efficient AWK variant with minimal memory footprint. |
| nawk | Traditional New AWK, preserving historical behavior for legacy scripts. |
| ag (The Silver Searcher) | Fast recursive searcher often replacing ack. |
| pt (The Platinum Searcher) | Newer recursive grep alternative with multithreading support. |
| ack | Perl‑based source‑tree searcher, maintained and optimized for code patterns. |
| ugrep | Feature‑rich modern grep clone with extended regex support and performance tuning. |
| sift | Recursive search tool for large directories, optimized for developer workflows. |
Benchmarking Methodology
- To evaluate the performance of the search engines, the benchmark focused on two critical metrics: runtime and peak memory usage, which together represent the total resource footprint.
- Resource tracking was performed with
cgmemtime, an ideal tool for capturing peak memory consumption for a process group. - The benchmarking process was automated via
benchgab.awk(version 2026.02.10), a custom runner that handles warm‑ups, multiple test runs, and calculates statistical metrics as well as normalized parameters for comparative analysis. - Each benchmark sequence included one initial warm‑up run followed by 100 recorded runs.
- The full methodology, test environment, raw data, statistical analysis, and normalized performance metrics are published on Awklab.com.
Normalized Metrics
| Symbol | Meaning |
|---|---|
| RT | Normalized median runtime – execution time relative to the fastest implementation (baseline = 1.0). |
| PM | Normalized median group peak memory – peak memory relative to the implementation with the lowest memory footprint (baseline = 1.0). |
| F | Resource Footprint – calculated as RT × PM. Lower values indicate a more efficient use of system resources to complete the same task. |
The normalized BEHILOS benchmarks were evaluated using Pareto‑frontier analysis (as previously applied in my AWK benchmarking study). To visualize search‑engine performance, the normalized values were plotted on a two‑dimensional coordinate system:
- x‑axis: Normalized runtime (RT)
- y‑axis: Normalized peak memory usage (PM)
The ideal point is at (1, 1), representing an implementation that is simultaneously the fastest and the most memory‑efficient. A logarithmic scale was applied to improve visibility of implementations clustered near the ideal point.
Graph: The Pareto frontier of search tools tested in the BEHILOS Benchmark, visualizing the optimal trade‑off between execution speed and memory footprint.
Results Overview
The normalized BEHILOS results clearly separate the tested tools into distinct performance tiers when runtime and peak memory usage are considered jointly.
Overall Winner
grepis the overall winner of the BEHILOS Benchmark.- Fastest implementation (RT = 1.00)
- Near‑minimum memory baseline (PM = 1.18)
- Minimum resource footprint (F = 1.18)
grep and mawk define the Pareto frontier.
Pareto Frontier
| Tool | RT | PM | F |
|---|---|---|---|
| grep | 1.00 | 1.18 | 1.18 |
| mawk | 1.83 | 1.00 | 1.83 |
grepis the fastest while staying very close to the minimum memory baseline.mawkachieves the lowest peak memory usage with only a modest runtime penalty. Neither tool can improve in one dimension without degrading the other, placing both on the Pareto frontier and representing the optimal trade‑off envelope for this workload.
Near‑Frontier Tools
| Tool | RT | PM | F |
|---|---|---|---|
| rg | 1.12 | 1.31 | 1.47 |
| ugrep | 1.20 | 1.38 | 1.66 |
| ag | 1.45 | 1.55 | 2.25 |
| gawk | 1.58 | 1.62 | 2.56 |
These tools are dominated by the frontier but remain reasonably close to it. Their normalized distance and F values indicate that they are not optimal for this specific task, yet their performance characteristics are still competitive. This reflects design choices favoring richer feature sets, broader file handling, or more general workloads rather than minimal footprint.
Outliers
| Tool | RT | PM | F |
|---|---|---|---|
| pt | 2.30 | 2.10 | 4.83 |
| sift | 3.10 | 2.45 | 7.60 |
| ack | 4.20 | 2.80 | 11.76 |
| nawk | 8.00 | 1.36 | 10.88 |
These tools lie far from the Pareto frontier. Their high normalized runtime and peak memory usage result in very large F values, indicating poor efficiency for this narrowly defined benchmark. They incur significant overhead relative to the simplicity of the BEHILOS search.
nawk Detail
Although nawk exhibits a low peak memory footprint (PM = 1.36), its slow runtime (RT = 8.00) places it well outside the efficient region. In addition, it showed the highest peak‑memory jitter among all tested tools, which further degraded its overall performance.
Takeaway
The BEHILOS benchmark demonstrates that, despite the proliferation of modern, feature‑rich search tools, the classic Unix grep remains the most efficient solution for a simple, character‑class‑only search. Its combination of speed and modest memory usage gives it the lowest overall resource footprint, reaffirming the timeless value of well‑engineered, focused utilities.
Conclusion
The Pareto analysis highlights that tools optimized for minimalism and predictability dominate this benchmark, while more feature‑heavy searchers pay a measurable cost in both runtime and memory.
The BEHILOS grep, despite originating as a historical anecdote from the early days of Unix, turns out to be an effective micro‑benchmark. Its simplicity isolates the core costs of regex matching, process startup, and memory allocation without confounding factors such as filesystem traversal or complex I/O patterns.
This benchmark shows that, for low‑footprint text searches, decades‑old design principles still matter. Classic Unix tools like grep, along with lean implementations such as mawk, remain hard to beat when efficiency is the primary goal. Modern search engines deliver powerful features and excellent performance for real‑world workloads, but those advantages are not free.
The BEHILOS Benchmark does not aim to crown a universal “best” search tool. Instead, it demonstrates how a minimal, well‑chosen workload can expose fundamental trade‑offs between speed, memory usage, and stability—and why even a small historical footnote can still teach us something meaningful about performance today.