Understanding the core mechanics of ThingsDB
Source: Dev.to
Query Handling
When a query is sent to a ThingsDB node, the node first checks its own status.
- AWAY mode – The node does not process the request locally; it forwards the query to another active node in the cluster, ensuring the client never experiences a blocked connection.
If the node accepts the query, it checks its internal cache:
- Cache hit – The exact same query has been seen before, so ThingsDB skips compilation and uses the pre‑compiled version.
- Cache miss – The query is new, and ThingsDB starts the compilation process.
During compilation, the engine determines whether the query requires a change.
Read‑only Queries
If no data changes are required, the query executes immediately on the node. No other nodes are involved, making read operations very fast.
Procedures
Procedures are always pre‑compiled. ThingsDB instantly knows whether they require a change, shaving off valuable microseconds.
Change Handling
If a query needs to modify data, ThingsDB initiates a “battle” for the next Change ID:
- It looks at the current global Change ID and proposes the next increment (Current + 1).
- Through a quorum‑based consensus, the majority of nodes must agree on this ID.
When consensus is reached, the change is processed. All modifications to the collection are tied to this specific ID and synchronized across the cluster, guaranteeing that every node handles changes in the exact same order and maintaining perfect data consistency.
Futures
ThingsDB uses Futures to handle complex logic. If a query contains one or multiple futures that require changes, each future can receive its own Change ID. This optimization:
- Prevents the system from locking up during external module calls.
- Allows heavy “change” logic to be isolated from the rest of the procedure or query.
Strategy tip:
If a procedure is mostly read‑only but occasionally needs to write data, wrap the write portion in a future. This prevents the entire procedure from generating a global change on every call (see the example in the ThingsDB book).
Data Persistence
ThingsDB employs a dual‑mechanism for persistence:
- Full State Dumps – Snapshots of the entire data state at a specific point in time.
- Archive Files – A continuous log of every individual change.
Node Boot and Recovery
When a node boots, it follows a strict recovery path:
- Load the last Full State Dump.
- Identify the last Change ID from that dump and replay all subsequent changes from the Archive Files.
- Wait for a peer node to enter Away mode. The booted node reports its last processed Change ID, and the Away node syncs the missing delta.
If a node is far out of sync (or a brand‑new node is added), a Full Synchronization occurs, transferring the entire state dump from scratch.
Away Mode
Away mode is ThingsDB’s mechanism for maintenance without downtime. While a node is in Away mode, it performs heavy‑lifting tasks such as:
- Full State Dumps (snapshots)
- Garbage Collection (memory management)
- Scheduled Backups
- Synchronizing peer nodes
During Away mode, the node still communicates with clients but forwards their queries to keep the system responsive. Changes are collected in the background and applied just before the node leaves Away mode and rejoins the cluster. Only one node can be in Away mode at a time, ensuring the rest of the cluster remains available for primary tasks.
Summary
By decoupling blocking maintenance tasks from client requests and using a sophisticated consensus mechanism for changes, ThingsDB delivers a high‑performance, highly resilient database experience.
Next up: the client side.