Pause · scale · drain · replay
StreamLake live sandbox
Pause a pipeline stage to create backpressure, scale workers to drain the backlog, and replay failed events from the dead-letter queue.
Everything is clickable. It runs in your browser with sample data and resets on refresh.
Healthy
Events processed
48,210,332
↗ 9.1%since midnight
Delivered now
11,800/s
of 11,800/s ingest
End-to-end lag
0s
target < 5s
Running
Pub/Sub ingest
events-prod topic
11,800evt/s
Backlog0
Running
Dataflow transform
Beam · enrich + dedupe
11,800evt/s
Backlog0
Workers6
Running
Schema validation
Avro registry check
11,800evt/s
Backlog0
Workers4
Running
BigQuery sink
streaming inserts
11,800evt/s
Backlog0
Workers4
Delivered to BigQuery
events per second, live04812162023
Production stack:GCPPub/SubDataflowBigQueryTerraformApache Beam