Thanks, both tips are good advice. One note on the coverage search for anyone reusing it: the raw-side count needs the same filter as the summary branch (the same where/case condition). If you count all raw events of the sourcetype, coverage will always look low even when nothing is lost. To your question: I tested it today. The panic is not limited to adding a branch. Reordering and removing a stats branch trigger it too. Setup: same pipeline (5 stats branches), Edge Processor 1.12.4 on two nodes, Ubuntu 24.04.5. Before each test I ran the original pipeline for 5 minutes so state files existed, then deployed the change without clearing statsStorage and sent load for 15 minutes. Reorder (swapped two nginx stats branches, nothing else changed) - 28 panics on the active node, about 2 per minute (one per flush): "index out of range [2] with length 2" and "[4] with length 4" - The two swapped branches kept only 2.7% and 2.9% of their events. The other stats branches stayed around 96%. Remove (deleted the pan_deny stats branch and sent that traffic to raw) - 12 panics on the active node, about 1 per minute: "index out of range [1] with length 1" - Every stats branch after the removed one lost data: 6.9%, 12.6% and 28.8% coverage. Branches before it were fine. The state files seem to be named by operator index (local-N.db (N = operator index)). After a reorder, the same index now belongs to a different stats definition. After a removal, every later operator shifts down by one and opens its neighbor's old file. Each panic drops the whole batch, not just the bad row. Only the node receiving traffic panics, so with a VIP the idle node will hit the same problem after a failover. Stack (same for both cases): aciesProcessor.processorRecover (processor.go:474) runtime.goPanicIndex PersistentSqliteDirectAggregationStore.GetAggregations (stats_direct_aggregation_store.go:141) TieredDirectAggregationStore.GetAggregationsBatch (:407) statsFunction.processDirectStats (stats.go:928-929) statsFunction.Process (stats.go:759) One more thing about the workaround: the order matters. Right after a restart, a node can run for a while with its cached old config and recreate the old .db files. What worked for me: 1. Deploy the pipeline change. 2. Wait until every node has picked up the new config. 3. On each node: stop the service, move statsStorage aside, start the service. 4. Check statsStorage again; if old files came back, repeat step 3. After that: 0 panics on both nodes.
... View more