Edge Processor sits in one of the most consequential places in any Splunk pipeline: between your data sources and the destinations they feed, whether that's a Splunk indexer, an Amazon S3 bucket, or Azure Blob Storage. When something in that path fails — an indexer reboots, a network link flaps, a destination throttles — every operator asks the same question:
Did I lose anything?
This post walks through how Edge Processor (EP) processes data end-to-end, then takes a deep look at the durable queue that backs every destination. If you've read our Validated Architecture for Edge Processor, you already know that EP queues data on disk per destination. This post is for readers who want to understand how it actually works: what the on-disk format looks like, how retries behave, and where the boundaries are.
The 30-second tour
Before we get into the queue, here's the path a single event takes through an Edge Processor instance:
The path an event takes through Edge Processor's components
Two things worth knowing up front:
Pipelines run in memory; destinations are backed by disk. The transformation work happens in memory. The moment an event leaves the pipeline and is bound for a destination, it's written to a disk-backed queue.
One queue per destination, shared across pipelines. Three pipelines all sending to the same S3 bucket share one queue. A pipeline sending to a Splunk indexer gets a separate queue. When a destination goes down, only its queue fills — the others keep flowing.
Receivers: how data gets in
Edge Processor accepts data over three protocols:
Splunk-to-Splunk (S2S): the protocol Splunk universal and heavy forwarders use to forward data.
HTTP Event Collector (HEC): the standard Splunk HTTP ingestion protocol.
Syslog (TCP/UDP, RFC 3164/5424): used by network devices and various log sources.
Pipelines: where SPL2 runs
Pipelines are where you write the SPL2 that filters, masks, enriches, and routes data. From a reliability standpoint, the key thing to know is that the handoff to the destination queue is synchronous. When a pipeline finishes processing an event and hands it to its destination, that handoff blocks until the queue accepts it. Only then does the pipeline accept the next event from its receiver.
The boundary of possible data loss is just the small window of in-memory work in the pipeline itself, and only in the case of an ungraceful shutdown. On a graceful shutdown, the pipeline flushes its in-flight work before exiting.
The durable queue: a write-ahead log per destination
When an event leaves a pipeline, it's written to a per-destination write-ahead log — a database technique where every change is written to an append-only file on disk before being processed, so data is preserved if the process is interrupted. We'll abbreviate it WAL from here on. This is the "durable" in "durable queue.”
What's on disk
The cursor readIndex and writeIndex are tracked in memory and segment files are written as the durable record on disk
The WAL is split into fixed 32 MB segment files, named by their starting index in hex ( 0000000000000000.bin , 0000000000000906.bin , and so on). Each event is written to its segment with a small header that includes its index, so the file itself is self-describing — the segments are the durable record of what's been received.
Alongside the segments lives a small cursor file that records the read and write indexes plus a checksum. The queue maintains its read and write indexes in memory at runtime and persists them to the cursor file when the process exits cleanly. For performance reasons, the cursor file is only written to on shutdown rather than on every event. Segment recovery rebuilds the right state if the cursor is stale.
The total queue size is configurable in gigabytes — default 5 GB per destination — which translates to a fixed maximum number of segments. When older segments have been fully delivered, they're cleaned up. When the queue grows, new segments are allocated until the configured cap.
The write path
When the pipeline hands a batch of events to the queue, two things happen:
The batch is serialized and written to the current WAL segment, with a per-event header that records its index.
If the segment is full, a new segment is allocated.
The handoff doesn't return until the write succeeds, and the pipeline doesn't move on until the handoff returns. Slowness in the queue propagates back to the receiver and ultimately to the sender.
Recovery on restart
Because the cursor file is written at shutdown rather than continuously, EP's recovery mechanism relies on the segments themselves. When the queue starts up, it walks the segment with the highest starting index, reading event headers in sequence until it finds either the end of the file or a header that doesn't match the next expected index. That tells the queue exactly where the write index left off, no matter how the previous process exited.
The read index is recovered with best-effort semantics. Once the write index is established from the segment scan, the queue uses the read index stored in the cursor file as-is and trusts it as the last known delivery position. If the cursor's read index exceeds the recovered write index (due to stale or corrupted cursor) or if the cursor file doesn’t match the checksum, the queue starts from the beginning of the oldest available segment and replays from there, which can produce duplicates downstream.
What happens when the queue fills up
This is the scenario every operator actually cares about: a destination is down, the queue starts filling, and now what?
When the WAL hits its configured cap, the handoff no longer returns immediately. Instead, it enters an exponential backoff — starting at 10 milliseconds and capping at one minute — and retries indefinitely until either a segment is freed or the EP instance is shut down.
EP does not silently drop data when the queue is full. It blocks, and that blocking is what propagates backpressure all the way up the chain:
A full queue propagates backpressure up the entire pipeline to the sender
Checking queue fill in real time
If you need a point-in-time answer to "how full is each queue right now?", the queue health extension exposes a lightweight HTTP endpoint that includes queue fill percentages directly in its response.
When USE_HEALTH_EXTENSION=true , EP starts an HTTP server on localhost:13133 . A GET /status?verbose=true returns JSON with the health of exporters and receivers. Each exporter entry includes a queue_fill_pct field (0–100) showing what percentage of the configured queue capacity is currently occupied:
{
"start_time": "2026-06-12T22:26:19.901774812Z",
"healthy": true,
"status": "StatusOK",
"status_time": "2026-06-12T22:27:42.210570677Z",
"components": {
"exporter:S2S/shared.pipelines.default_splunk_cloud_destination": {
"healthy": true,
"status": "StatusOK",
"status_time": "2026-06-12T22:26:19.967057594Z",
"queue_fill_pct": 9.06
},
...
"receiver:S2S/splunk.ingest.edge_processor.all_data_raw": {
"healthy": true,
"status": "StatusOK",
"status_time": "2026-06-12T22:26:19.984922984Z"
},
}
}
Note the ?verbose=true query parameter — without it, /status returns only the top-level aggregate with no per-component breakdown. Component keys follow the format "<component>:<protocol>/<dataset-name>" .
A value approaching 100 means the destination is falling behind. Sustained high fill means the queue will eventually block the pipeline feeding it.
How retries actually work
Once data is in the WAL, consumer workers read batches and hand them to the destination's exporter. What happens next depends on the kind of error:
Success. The batch is committed and the WAL space is eventually reclaimed.
Error (network blip, 5xx, timeout). The exporter's own retry logic handles short-lived failures. If those exhaust, the worker logs a warning and stops. The batch is not committed — it stays on disk in the WAL and will be retried.
Graceful shutdown. When EP shuts down cleanly, in-flight consumer work is given time to complete and commit before the queue closes. Only if shutdown is forced before that window closes does a batch go uncommitted — in which case it stays on disk and is re-delivered on the next start.
Restarts are recoverable, and the cost of an ungraceful shutdown is potential duplicates.
What EP guarantees — and what it doesn't
Stage
Loss possible?
Notes
Durable queue write
No
Synchronous handoff; pipeline blocks until written.
Queued on disk
No
Every event on disk is delivered at least once; segment-scan recovery rebuilds state after any restart.
Delivery to destination
Yes, permanent errors only
Transient errors are retried. Permanent errors (HTTP 400/401-class) are discarded with a warning log.
In-memory pipeline
Yes, ungraceful shutdown only
Events in-flight at the moment of an ungraceful shutdown are lost. On a graceful shutdown, the pipeline flushes before exiting.
Startup with stale or corrupted cursor
No
If on startup the cursor file is stale/corrupted, the queue replays from the start of the oldest segment which may produce duplicates.
Once an event is past the pipeline, EP either delivers it or tells you in the logs that it gave up and why. It does not silently drop data to relieve backpressure.
What happens when an EP node goes down
The WAL directory lives on a persistent data volume, so segment files survive a process death. The supervisor restarts EP, the queue recovers using the segment scan and cursor recovery described above, and delivery resumes. Data that had already been written to the WAL before the termination is safe. The risk window is events that were in-memory mid-pipeline at the exact moment of an ungraceful shutdown.
Monitoring
The durable queue emits metrics you can use to monitor its status and health.
The full reference can be found here.
... View more