SC4S Dropping Events Going to Edge on Same Server

[daniel333]

All, 

Running into some trouble here with Syslog > SC4S > Splunk Edge. I think I have tuned every knob there is to be tuned but I am finding events are still being dropped. 

SC4S writing to a file, does NOT drop events. But sending to the Edge HEC interface did. Sending Syslog direct to Edge , no dropped events. Not seeing queues anywhere back up. Plenty of resources. Not really sure where to start. 

I followed the SC4S troubleshooting guide and we checked all the boxes there.  Any secrets docs or troubleshooting steps I should be looking into? 

Below is an AI summary of what has been done so far: 

```

SC4S → Edge Processor over HEC: a tiny, stubborn ~0.008% loss we could never tune to zero (and why we gave up on the design)

**Short version:** We ran SC4S (axosyslog) forwarding to a co-located Splunk Edge Processor via HEC on `http://localhost:8088`. Under UDP syslog load it dropped a small, maddening fraction of events — around **0.008%–0.2%** — that we could never drive to a repeatable zero, no matter what we tuned. We eventually proved the loss is inherent to pushing UDP (no backpressure) through syslog-ng's `http()` destination, and abandoned the SC4S→EP-over-HEC design. Posting the full knob list, the measurement method, and the lab-vs-prod result in case it saves someone else the two nights we spent.

---

## The setup

```text
syslog senders (UDP :514)
│
▼
SC4S (axosyslog 4.x, container3, podman, --network host)
│ classifies vendor, stamps index + sourcetype
│ forwards over HEC to a co-located Edge Processor
▼
Edge Processor (same box, HEC receiver on :8088, plaintext)
│
▼
S2S :9997 → indexer cluster
```

- Both SC4S and EP on one 8-core box, ~72% idle at our volume. Not resource-starved.
- The HEC hop is over **loopback** — SC4S `http()` → EP HEC on `localhost:8088`.
- We must support **UDP** ingest (device standard). This turns out to be central.

---

## How we measured loss (this is the part that matters)

We stopped trusting counters early — they lie in both directions (indexing lag makes good delivery look lossy; all-time counters hide per-run drops). The only number we believed:

1. **Sequenced load.** A generator sends exactly **N = 100,000** synthetic events at a fixed rate. Every event carries a unique run tag plus a zero-padded sequence number `seq00000000 … seq00099999`.
2. **Full settle.** Wait 90 seconds so all indexer peers have received and indexed everything. (Skipping this is how you manufacture fake loss — we did it a couple of times early and chased ghosts.)
3. **Count distinct sequence numbers that landed:**

```spl
index=* "seq-<runtag>"
| rex "seq(?<n>\d{8})"
| stats dc(n) as distinct count as raw
```

`100000 - distinct` = true loss. `raw - distinct` = duplicates (mattered — see below).

4. **Forensics.** Bin the sequence space into 100 buckets of 1,000 and count missing per bucket, to see *where* loss happens (burst vs steady, contiguous vs scattered).

> Gotcha: `... | stats count by n` silently caps at 50,000 rows, which makes everything above seq 50000 look "missing." Use binned `dc(n)`, never per-seq `stats` over 50k events.

---

## The single most useful test: does SC4S even have the events?

SC4S has a built-in archive/file destination (`SC4S_ARCHIVE_GLOBAL=yes`) that writes every received event to disk in parallel with HEC — no HTTP, no flow-control window. We turned it on and ran the sequenced 100k:

```text
sent: 100,000
archive file (disk): 100,000 distinct seq ← zero missing, zero dups
HEC (into indexes): 99,905 distinct seq ← ~95 missing
```

**SC4S receives, parses, and retains 100% of events.** The loss is **entirely** the SC4S `http()` → EP HEC delivery hop. That one test ruled out UDP-receive loss (already zeroed with kernel tuning), SC4S internals, and classification.

---

## Every knob we turned, and what each did

Baseline loss on the sequenced 100k: **~0.008%–0.2%** (noisy, never a clean repeatable zero). Measured by distinct-seq-landed.

| Knob | Change | Result |
|---|---|---|
| `net.core.rmem_max` (kernel UDP recv buffer) | 208 KB → 512 MB (+ SO_RCVBUFF, IW) | **Fixed the UDP *receive* loss → 0** (was ~2.3%, 49.5M rcvbuf errors). Separate problem, worth doing. |
| `net.core.wmem_max` (kernel TCP send buffer) | 208 KB → 512 MB (+ netdev_max_backlog, tcp_wmem) | Helped a bit — produced our only true 0.000% run — but not repeatably (next run 0.14%). Kept it; costs only memory. |
| SC4S HEC `workers()` | 1 → 20 → 40 | workers=1 was ~11× worse (0.89%). workers=40 best (~0.008%). But the workers sit ~idle (0.4% CPU, queued=0) — not the bottleneck. |
| `batch-lines` | 100 → 5000 | **Worse** (0.61%). Bigger batch = a bigger chunk exposed at the flow-control window when a flush stalls during a UDP burst. |
| `batch-lines` | 100 → 50 | **Worse** (0.20%). SC4S's small-batch default is the shallow minimum — both directions hurt. |
| Reliable disk-buffer (mem + disk) | enabled, 256 MB / 10 GB | No effect on this loss. `queued=0, memory_usage=0` when it drops — the buffer sits behind the queue; the drop is *before* the queue. |
| HEC URL | `localhost` → external NIC IP | No change (~0.04%). Rules out a loopback-stack artifact. |
| `useACK` (HEC indexer acknowledgement) | considered | Dead end — syslog-ng's `http()` dest doesn't do HEC ACK, and SC4S docs warn against it. |
| `response-action(400 => drop)` → `retry` + `retries(10)` | tried | **Worse** (0.9% → 2.2%). Retrying rejected batches consumed the workers and starved *fresh* traffic. The `400 => drop` default exists for a reason. Reverted. |

The two kernel-buffer changes are the only ones we kept. Everything on the SC4S send side was neutral or harmful.

---

## The fingerprint that told us it wasn't tunable

Sequence-gap forensics on a lossy run: ~60% of a 100k run was **literally zero loss**, and all the drops fell into a handful of **2–5 second windows** — contiguous bursts, not a uniform statistical rate. SC4S and EP logs were both **silent** in those windows (no error, no reconnect). `dst.http dropped` incremented while `queued=0` and `memory_usage=0`.

That is the signature of a **flow-control-window discard at the log path *before* the destination queue** — a sub-logging-threshold drain stall (a GC pause / scheduling hiccup / momentary EP HEC pause) that the flow-control window can't fully absorb in that instant. Because the source is UDP, syslog-ng **can't** apply backpressure — there's no sender to slow down — so it discards at the window instead of blocking.

Buffers, workers, CPU, and retries all live *downstream* of where the drop happens, which is why none of them reach zero.

---

## Lab and prod: same result

We built a clean lab clone (fresh podman SC4S, idle 16-core box, no real traffic, no other tenants) and ran the identical sequenced load: it dropped **0.15%** — same order of magnitude as prod. So the residual is **not** a prod-contention or noisy-neighbor artifact. It reproduces on an idle lab box. It's inherent to the transport, not the environment.

---

## Conclusion — why we abandoned the SC4S→EP-over-HEC design

After the two kernel fixes and ~nine send-side levers, the honest conclusion is: **you cannot tune the SC4S `http()` → EP HEC hop to a consistent 0% under UDP burst.** The residual is the UDP-has-no-backpressure property surfacing as a flow-control-window discard. Every lever either did nothing, made it worse, or was a band-aid over a statistical edge race.

What actually gets to zero (neither is a knob):

- **Archive/disk-buffer as a safety net** — proven to hold 100% on disk. It makes "we never *lose* an event" true even if HEC sheds, but it's a replay net, not a fix for the hop.
- **Get off HEC entirely** — end-to-end **TCP** (backpressure exists → the sender waits instead of anything dropping). In our case we ended up dropping SC4S and sending syslog **directly to the Edge Processor's own syslog receiver**, which sidesteps the SC4S `http()` destination completely.

---

## Questions for anyone who's run this

1. Has anyone driven the SC4S (axosyslog `http()`) → Edge Processor HEC hop to a *repeatable* 0% under sustained UDP burst? If so, what did it take?
2. Is there an `http()` setting that makes the destination **block/park into the reliable disk-buffer** instead of discarding at the flow-control window when the receiver stalls — without the retry-storm side effect we hit?
3. Anyone tuned the **Edge Processor HEC receiver** side (accept concurrency / receive window / max in-flight) rather than the SC4S send side? The EP docs we found only mention the sink send-queue.
4. Is SC4S → EP over **syslog/S2S** (bypassing HEC) a supported pattern, and if so how do you preserve SC4S's index/sourcetype routing on that transport?

Happy to share the sequenced-load generator and the exact SPL. Measuring by distinct sequence IDs (and checking SC4S's own on-disk archive as ground truth) is what finally made this honest — and what convinced us the design wasn't tunable, only replaceable. Thanks in advance.

```

[masonreed11]

Your testing already points strongly to the SC4S → HEC hop as the bottleneck. Since direct syslog to Edge is lossless, I’d avoid over-tuning http() and look at bypassing HEC or using a transport with backpressure. The archive test was a great way to prove the loss isn’t happening at SC4S ingestion.