topic Drop 9 of 10 Events (Sampling) in Getting Data In

Drop 9 of 10 Events (Sampling)

CMEOGNAD — Fri, 10 Mar 2023 09:34:20 GMT

Hi @ All Splunkynators

how to sample incoming (HEC) data?
I want get statistical data /events to save license volume, drop eg 9 of 10 of incoming events...

I look forward to your suggestions 🙂

Gegards - Markus

Re: Drop 9 of 10 Events (Sampling)

rnowitzki — Fri, 10 Mar 2023 13:16:50 GMT

Hi @CMEOGNAD ,

See my response to a similiar question. You could drop events with seconds *1 to *9, keep *0

It will not be exactly 9 of 10. But with something like that you could randomly reduce the volume.

Not sure if it makes sense with your data.

(or Cribl)

Re: Drop 9 of 10 Events (Sampling)

PickleRick — Fri, 10 Mar 2023 13:32:27 GMT

You can't do that directly in the splunk input phase. You can't keep "state" on event processing so you can't count events and keep track of which event you're processing at the moment. So there's no way to enforce the strictly "every 10th" rule. You can do somethig like time based as @rnowitzki suggested or try to implement some INGEST_EVAL logic to base your decision on random value (and check if it falls within some range) or - for example - message hash. But this will of course give you some percentage on average, not a strict 1 out of every N policy.