Getting Data In

Drop 9 of 10 Events (Sampling)

CMEOGNAD
Engager

Hi @ All Splunkynators

how to sample incoming (HEC) data?
I want get statistical data /events to save license volume, drop eg 9 of 10 of incoming events...

I look forward to your suggestions 🙂

Gegards - Markus

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You can't do that directly in the splunk input phase. You can't keep "state" on event processing so you can't count events and keep track of which event you're processing at the moment. So there's no way to enforce the strictly "every 10th" rule. You can do somethig like time based as @rnowitzki suggested or try to implement some INGEST_EVAL logic to base your decision on random value (and check if it falls within some range) or - for example - message hash. But this will of course give you some percentage on average, not a strict 1 out of every N policy.

0 Karma

rnowitzki
Builder

Hi @CMEOGNAD ,

See my response to a similiar question.  You could drop events with seconds *1 to *9, keep *0

It will not be exactly 9 of 10. But with something like that you could randomly reduce the volume. 

Not sure if it makes sense with your data.

(or Cribl)


--
Karma and/or Solution tagging appreciated.
0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...