Projects / Built · Case study
Making live trading and backtests run the same code
An architecture for running algorithmic crypto strategies on live exchange data, designed so that an algorithm cannot tell whether its market data is live or historical.
- Context
- Crypto trading platform
- My role
- Architect; led a team across frontend, backend, DevOps and QA
- Stack
- Kafka, gRPC, Protocol Buffers, Docker, Kubernetes or equivalent, Binance WebSocket
- Focus
- Live and backtest parity
The problem
A trading strategy has to be tested on history before it risks money, but if the backtest runs a different code path from production, it is testing something other than what will actually run. The platform’s core requirement was to make that divergence impossible, while still ingesting real-time exchange data at scale and handing it to many independently deployed algorithms without re-architecting ingestion each time one is added.
What I did
- Designed the event backbone: market data lands in topics partitioned by data type, not by symbol, so storing it and feeding strategies scale and fail independently through separate consumer groups.
- Designed an application-level router with a subscription registry, so each algorithm receives only the symbols and data types it asked for and the platform can grow in algorithms and users without touching ingestion.
- Set the standard that backtests replay stored history through the same event stream used in production, so no algorithm carries separate backtest logic.
- Chose gRPC bidirectional streaming between the runtime and each algorithm container: one persistent, language-neutral channel for events going down and for subscriptions, order intents and heartbeats coming up.
- Led the cross-functional team on delivery and architecture decisions.
One algorithm, two data sources
The full system
Five abstractions that make it hold
- Canonical market event
One event shape for trades, book updates, klines and funding, whether it came from a live socket, stored history or a synthetic source.
- Algorithm interface
Handlers such as on-trade and on-kline. The algorithm never branches on live versus backtest.
- Trading clock
Wall-clock time in live mode and replay-event time in a backtest. Algorithms never read the system clock.
- Order intent
The only way an algorithm expresses action. The risk engine and either live execution or the simulator decide what happens next.
- Platform SDK
Hides Kafka, gRPC, serialization and reconnects, so an algorithm developer implements a strategy and nothing else.
What I chose, and what lost
- Kafka as the backbone
Over direct service WebSockets, SSE, REST or gRPC alone. None gives several independent readers of one stream their own offsets, so a slow consumer could not be isolated.
- Topics by data type
Over one topic per symbol, which would multiply topics with the symbol count into the hundreds or thousands. Symbol is the partition key instead, so same-symbol events stay ordered while symbols still parallelize.
- A router in application code
Kafka knows topics and offsets, not “algorithm A wants BTC and ETH”. That mapping changes often, so it belongs in application state and keeps Kafka a generic backbone.
- gRPC bidirectional streaming
Over WebSocket, which would leave framing, reconnects and routing to build, and over SSE, which cannot carry subscriptions or order intents back. Each algorithm connection gets its own bounded queue, so a stalled algorithm degrades alone.
- A replay engine
Over exposing the historical database to algorithms. Algorithms consume events, not schemas, and replay keeps ordering, pacing and the deterministic clock under control.
Keeping it up when parts fail
- A slow algorithm
Each algorithm connection has its own bounded queue, so a stalled algorithm cannot block the others. Lag is measured and alerted on, and the algorithm can be disconnected and restarted.
- Dropping events
The policy depends on the event type. Trades may be coalesced for some strategies, order-book deltas need strict sequencing, and account and trading events are never silently dropped.
- An algorithm crash
The runtime detects it and updates the registry. Every other algorithm keeps running.
- A runtime instance dies
Runtimes share one consumer group, so Kafka reassigns the partitions to another instance, which rebuilds its subscription and routing state.
- Exchange disconnect
The gateway reconnects, resubscribes and validates stream continuity, so a gap is detected rather than silently passed on.
- Kafka producer errors
The gateway never discards data silently. It buffers briefly, reconnects, alerts, and recovers from exchange state where needed.
- Storage falling behind
Persistence and strategies read in separate consumer groups, so if strategy consumption lags, writing history continues.
Rules the design enforces
- Algorithms never consume exchange messages directly. Data reaches them as a platform event.
- Algorithms don’t know Kafka. The SDK hides it.
- Live and historical data use the same event model.
- Algorithms never query the historical database. The replay runtime provides history.
- Algorithms never submit exchange-specific orders. They emit an order intent.
- Topics represent data types, not individual symbols, and symbol is the partition key where ordering matters.
- Algorithm subscriptions are application-level state held by the runtime, not by Kafka.
Limits the design states openly
- Kafka is not automatically the lowest-latency path. If true high-frequency latency were ever needed, the market-data path would have to be benchmarked against an in-memory bus. It was an explicit non-goal.
- Order-book strategies need stricter guarantees than trade ticks, such as sequence and gap detection.
- Running one algorithm binary removes software divergence between live and backtest, but slippage, partial fills and exchange matching still cannot be perfectly simulated.