The tick store trading desks run on, under your dashboard.
BDOBB's backend stack, openbb-docker, keeps the live trading session in kdb+ and its language q. KX describes kdb+ as the engine "the world's fastest trading firms have run on for 30 years", and says its surveillance product built on kdb+ is "used extensively in high-frequency, electronic trading". Every update is appended to a tickerplant-style log before it reaches memory. When the process restarts, q replays that log, so the cache comes back with every update it had accepted.
kdb+tick, the layout KX documents for real-time data.
A "vanilla" tick setup has a tickerplant logging to disk and publishing to an in-memory real-time database (RDB). At day end, the RDB's data is saved as another day in the historical database (HDB). Click any box to see what it does in the standard design and what plays that role in this stack.
Tickerplant
Same durability contract, fewer moving parts.
The stack keeps the parts of kdb+tick that make a restart lossless: an append-only log, replay with -11!, and a daily roll. It folds the tickerplant and the RDB into a single q process, and it sends finished days to Delta Lake rather than to a q HDB.
| Concern | Standard kdb+tick (KX) | BDOBB backend stack |
|---|---|---|
| Write path | The tickerplant's .u.upd publishes the message, writes it to the log, and increments .u.i (tick.q). | upd is wrapped once. Each update is appended to the log, then inserted into trades and pushed to WebSocket subscribers. |
| Log file | One file per day, <dir>/<schema><date>, held in .u.L. KX advises a fast local disk. | /data/logs/tlog_YYYY.MM.DD, one O(1) append per update. This replaced an earlier full-table mirror written every 5 minutes. |
| Recovery | The RDB asks the tickerplant for the schema, the log location and the message count, then replays that many chunks with -11! (r.q). | At boot, q replays every log on disk, oldest first, with -11!. It replays only into an empty table, so reloading the script into a running server doesn't duplicate rows. |
| Day roll | .u.endofday calls each subscriber's .u.end and starts a new log. | A 60-second timer starts a new log when the UTC date changes and deletes logs older than 7 days. |
| History | The RDB writes a date partition to the HDB at end of day. | A sidecar writes each finished day to Delta Lake, one table per symbol and date, overwriting the whole table so a re-run is safe. Charts read completed periods from that tick vault. |
| Real-time analytics | RTEs compute on every update, such as order books and latest prices, and should finish within the publish interval. | Served over WebSocket: a 500-tick snapshot on subscribe, OHLCV and VWAP bars of any width, anchored VWAP by time or trade count, as-of joins and memory stats. |
| Exposure | Gateways route queries and return sensible errors. | q listens only on 127.0.0.1. The HTTP handler serves two read-only paths and replaces q's default handler, which can run code. |
Kill the process.
Watch the log bring it back.
This simulation runs in your browser and mirrors the boot sequence in openbb-docker's persist.q. Let a few ticks arrive, kill q, then restart it. Then kill it again, corrupt the tail of the log, and restart to see KX's standard repair. Counts and byte sizes are illustrative.
tradestlog_2026.10.13The drill covers what the log covers: updates q accepted before it stopped. What a feed does while q is down depends on the feed, so the drill pauses it.
How kdb+ shops bring a real-time database back.
These are KX's own procedures, from its kdb+tick source and its documentation on log files and disaster recovery.
Ask the tickerplant where to start
A restarted RDB is empty. Its first act is a single synchronous call that subscribes and returns the table schemas plus the log's message count and file name. KX's r.q does it in one line:
.u.rep .(hopen `$":",.u.x 0)"(.u.sub[`;`];`.u `i`L)";Replay exactly that many messages
.u.rep creates the tables, then streams the log through -11!. Each chunk is evaluated as it is read, so replay doesn't load the whole file into memory. KX's architecture guide says the log "is used for recovery: if the RDB has to restart, the log file is replayed to return to the current state."
.u.rep:{(.[;();:;].)each x;if[null first y;:()];-11!y;system "cd ",1_-10_string first reverse y};Let queued updates drain, and route queries elsewhere
Updates published during the replay wait in the TCP buffer and then in tickerplant memory, visible in .z.W. In extreme cases this can cause a wsfull error. Replay can take minutes or more, so the gateway keeps sending queries to the secondary until the primary is ready (KX, disaster-recovery planning).
If the tail is corrupt, keep the valid part
A plain replay of a damaged file signals 'badtail. -11!(-2;file) returns the number of valid chunks and the length of the valid part. You then rebuild a good log from those chunks, or copy the first N bytes with head -c. KX notes this "does not fix the corrupted section, only removes" it (KX, log files).
q)-11!(-2;logfile)
26
35634
goodfile:hsym `:good.log; goodfile set ();
goodfilehandle:hopen goodfile;
chunks:first -11!(-2;`:bad.log);
.z.ps:{goodfilehandle enlist x};
-11!(chunks;`:bad.log);
system"x .z.ps"; hclose goodfilehandle;persist.q/ wrap upd exactly once: append to the log, then the original insert+publish
if[not `updCore in key `.; updCore:upd];
upd:{[t;d] if[not null .tlog.h; .tlog.h enlist(`upd;t;d)]; updCore[t;d]};
/ boot replay: every log still on disk, oldest first. Hot-load guard.
if[not count trades;
{[f] -1 "tlog: replaying ",string f; -11! ` sv .tlog.dir,f} each asc key .tlog.dir];The log handle is still null during the replay, so replayed updates are not written to the log a second time. The day's log is opened only after the replay finishes.
What production kdb+ sites add on top.
KX's disaster-recovery white paper describes the practices capital-markets sites layer on top of log replay. Each card shows where this stack stands today.
Hot-hot secondary
Not in this stackA full mirror of the feedhandler, tickerplant and subscribers on separate infrastructure, with its own file system, that can take over when anything critical goes down.
Failover on disconnect
Not in this stackSubscribers get a closed-connection callback in .z.pc and use it to switch to the secondary tickerplant. Afterwards, reconcile the primary and secondary.
Dedicated log drives
Your hardwareTickerplant logs belong on fast, dedicated drives with no contention, to keep publishing fast and replay short. Here, that depends on where you mount /data.
Back up logs and the sym file
PartlyKX stresses backing up tickerplant logs as well as the database, and the HDB sym file in particular. Here, logs stay on disk for 7 days and finished days are written to Delta Lake, which has no q sym file.
Lossless restart
In the stackReplay the log at startup to rebuild memory exactly. This is the core of kdb+tick, and here it covers every day still on disk, not only today.
Capacity headroom
ReferenceKX cites up to 4.5 million streaming records per second per core, or 10 million with batching. Its RDB sizing rule is expected messages × schema cost × a flexibility ratio, commonly 1.5.
'badtail error stops the boot replay until someone runs the KX repair. The tickerplant and the RDB run as one process, so a crash takes down the write path and the cache together.
An addition to your OpenBB setup, not a replacement.
- A live session you don't lose. The intraday cache survives a restart, an upgrade or a container rebuild, because the log on disk is the record of every update.
- Trading-desk primitives on the wire. Bars of any width, VWAP from any anchor, and as-of joins are computed where the ticks live, rather than in the browser.
- History without a q licence for every reader. Finished days go to Delta Lake, an open Apache-2.0 format, which any reader can query.
- Your licence, your binary. The repository ships no kdb+. You install kdb-x or kdb+ Personal Edition from KX and mount your own licence file.
live profile runs the live-grid sidecar with this kdb+ cache, vendored as openbq-kdb 10.0.0 from Adventures-OpenBQ/openbq-kdb. Finished days go to Delta Lake on RustFS, and live-grid reads them back. Live checks against a running stack are still to be run on the Mac.
Sources
- KX, Architecture: feedhandler, tickerplant, TP log, RDB, RTE, HDB and gateway roles and best practices.
- KX, kdb+tick alternative architecture: the "vanilla" tick setup.
- KX, tick.q and r.q reference pages, and the kdb-tick r.q source.
- KX, Using log files: logging, recovery and replication:
-11!, partial replay,badtailand repair. - KX (Stewart Robinson), Disaster-recovery planning for kdb+ tick systems.
- KX, kx.com, About KX and KX Surveillance: use of kdb+ in trading.
- bdobb-dev/openbb-docker (private):
kdb-ws/startup.q,kdb-ws/persist.qv2 andkdb/README.md, main at c83446c. - Adventures-OpenBQ/bdobb (private) v10.0.0 release notes and
OPEN-QUESTIONS.md: the live grid, the openbq-docker v10liveprofile and openbq-kdb 10.0.0.
kdb+, q, KDB-X and KX are trademarks of KX Systems, Inc. BDOBB and SecretoftheUniverse.com LLC are not affiliated with or endorsed by KX. The kdb+ binary and licence come from KX under KX's terms.