Log Aggregation Pipeline Simulator
Follow production logs from applications through collection, parsing, filtering, buffering, sharded indexing, and search. Trigger traffic spikes, parser failures, and slow indexing to see where backpressure builds and logs can be lost.
Category: Observability
What You Will Learn
- How application logs move through collectors, processors, buffers, indexes, and search
- Why parsing and filtering happen before durable indexing
- How bounded buffers absorb short bursts but eventually overflow
- How traffic spikes, parser mismatches, and slow indexing create different failure signals
- How document routing distributes indexed logs across search shards
- Why filtered, rejected, and dropped events never appear in search results
Topics covered: logs, observability, elasticsearch, fluent-bit, parsing, backpressure, monitoring, sre, educational, interactive
// simulator
Log Aggregation Pipeline Simulator
Follow production logs from applications through collection, parsing, filtering, buffering, sharded indexing, and search. Trigger traffic spikes, parser failures, and slow indexing to see where backpressure builds and logs can be lost.
Production log pipeline
Capacity is keeping up and accepted logs are progressing toward search.
Cycle 1 · next stage
Select a stage to inspect what it does.
Next: Apps & hosts
Last step: Ready. Generate a batch to start the pipeline.
Generated
0
all incoming
Buffered
0
waiting to index
Indexed
0
searchable
Rejected
0
parser mismatch
Dropped
0
lost to overflow
Stage inspector
Apps & hosts
Applications, containers, and hosts emit raw log events.
Operator signal: Watch the incoming rate. A sudden increase is often the first sign of an incident.
Grok runs a regex on every line, so the processor handles fewer logs per cycle. Changes require a fresh run.
Reduces storage volume before indexing.
Three primary shards
Events are distributed by document id. Balanced bars mean balanced indexing work.
How a log pipeline stays reliable
1. Collect close to the source
Lightweight agents tail files or container output and forward events. Local queues keep a brief network problem from immediately losing logs.
2. Shape before storage
Parsers extract searchable fields, enrichment adds context, and filters remove known noise. Reject counters expose format changes before they become blind spots.
3. Buffer the mismatch
Collection and indexing rarely run at exactly the same rate. A durable bounded buffer absorbs bursts while backlog alerts give operators time to restore capacity.
What to alert on in production
Monitor input rate, queue age and depth, parser rejection rate, buffer utilization, dropped events, indexing latency, and shard balance together. A single healthy search query does not prove the pipeline is complete: missing logs may have been rejected long before they reached the index.
Overview
A log pipeline moves events from the hosts that write them to a store where people can search them, and every hop has its own capacity. This simulator models that chain as six stages: apps and hosts, a Fluent Bit collector, a parse and filter step, a durable buffer, a search cluster and the explore view where you query logs. Each tick advances the pipeline by one stage, so you can see exactly where events pile up. Four scenarios cover the common ways pipelines break. A healthy flow drains normally. A traffic spike outruns the collector at the edge. A parser mismatch, caused by a deployment that changes the log format, makes structured parsing reject events. Slow indexing leaves the search cluster unable to keep up, so the buffer absorbs backpressure until it fills and starts dropping the oldest logs. The lesson is that a buffer turns a slow downstream into a queue instead of data loss, but only until it is full, which is why buffer depth is one of the first things to alert on.
Try this
- Run the "Healthy flow" scenario, then switch to "Slow indexing" and watch the durable buffer grow until it reaches capacity and drops the oldest logs.
- Pick "Parser mismatch" and inspect the parse stage to see how many events structured parsing rejects after the log format changes.
- Step the pipeline one tick at a time and compare how a "Traffic spike" creates pressure at the collector instead of at the search cluster.
Frequently asked questions
- What is backpressure in a log pipeline?
- Backpressure is what happens when a downstream stage cannot accept data as fast as it arrives. Events queue up behind it, and without a buffer the upstream stage has to slow down or drop data.
- Why put a buffer between the collector and the search cluster?
- A durable buffer, such as Kafka or a disk-backed queue, absorbs bursts and slow indexing so logs wait instead of being lost. When it fills up, the pipeline has to pause upstream ingestion, reject new events or drop events, depending on its overflow policy.
- Why do logs disappear after a deployment?
- A common cause is a changed log format. If the parser expects the old structure, it rejects or misroutes the new events, so they never reach the index in a searchable form.
- What should I alert on in a log pipeline?
- Buffer depth and its growth rate, dropped or rejected events, parse failures, and the lag between when an event is written and when it becomes searchable.
Try next
// simulator
Prometheus Query Builder (PromQL Playground)
Learn Prometheus Query Language (PromQL) with an interactive playground. Master label filtering, rate calculations, aggregations, and histogram quantiles with real-time query execution.
// simulator
Preview Environment Simulator
See how a pull request becomes a temporary copy of your app, complete with a private URL, safe test data, review checks, and automatic cleanup.
// game
Heroku-Style Name Generator
Generate names the way Heroku, Docker, Kubernetes, and petname do, see how the word lists combine, and learn when the birthday problem makes random names collide.