Lighter Liquidations

Open sourceBackend service

Backend service for Lighter.xyz that ingests perpetuals liquidation events over WebSockets, deduplicates noisy replay batches, aggregates notional USD into per-minute buckets in Upstash Redis, and serves a rolling 24 hour liquidation total over HTTP.

View Source Code

time-series bucket size: 1 min
rolling liquidation window: 24h
isolated processes: worker + API: 2

Tech stack

Node.js
WebSockets (Lighter streams)
Upstash Redis
Minute-bucket time series
HTTP REST API

Overview

Lighter Liquidations is a real-time data pipeline that tracks liquidation activity across Lighter.xyz markets and maintains a rolling 24-hour liquidation metric. The system is designed to handle noisy WebSocket streams, eliminate duplicate events, and aggregate high-frequency data into a query-efficient time series.

It follows a split architecture: a persistent worker ingests and processes streaming data, while a stateless HTTP API serves pre-aggregated results. Redis acts as the aggregation layer using minute-level buckets with TTL-based retention.

What it does

Consumes liquidation events from Lighter WebSocket streams in real time
Deduplicates repeated or batched events to prevent inflated metrics
Aggregates liquidation volume into minute-level time buckets
Maintains a rolling 24-hour liquidation total using Redis TTL
Exposes a lightweight HTTP API for dashboards and integrations

Technical breakdown

System architecture

The system is split into two independent processes: a long-running worker and a stateless HTTP server. The worker maintains WebSocket connections, processes incoming liquidation events, and updates Redis in real time. The API server reads aggregated data from Redis and exposes it via REST endpoints.

Ingestion and data layer

WebSocket ingestion is handled using Node.js with persistent connections and reconnection logic. Redis (Upstash REST) is used as a time-series store where each minute acts as a bucket. Deduplication logic ensures idempotent aggregation by filtering repeated events within batch windows. The rolling 24-hour metric is computed dynamically by summing active buckets.

Technical challenges

01
Deduplication of noisy WebSocket streams
Lighter streams often send repeated or batched liquidation events, which can lead to overcounting. A deduplication strategy was implemented by comparing key fields within short time windows, ensuring idempotent aggregation without dropping legitimate events.
02
Designing a rolling time window
Instead of storing a single aggregate value, liquidation data is stored in minute-level buckets with a 25-hour TTL. This allows efficient computation of a rolling 24-hour metric while keeping memory usage bounded and predictable.
03
Separation of ingestion and serving layers
Running WebSocket ingestion and API serving in the same process created stability issues. Splitting them into worker and server processes ensured better fault isolation and allowed independent scaling and restarts.
04
Handling WebSocket reliability
Frequent disconnects and network instability required implementing reconnection logic and ensuring no data corruption during reconnect cycles. The system prioritizes consistency over completeness, avoiding incorrect spikes in metrics.

Who it is for

Built for analytics platforms, trading dashboards, and developers who need accurate real-time liquidation data from Lighter.xyz without handling raw WebSocket streams or dealing with duplicate event noise.