Getting Data In

Drop 9 of 10 Events (Sampling)

CMEOGNAD
Engager

Hi @ All Splunkynators

how to sample incoming (HEC) data?
I want get statistical data /events to save license volume, drop eg 9 of 10 of incoming events...

I look forward to your suggestions 🙂

Gegards - Markus

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You can't do that directly in the splunk input phase. You can't keep "state" on event processing so you can't count events and keep track of which event you're processing at the moment. So there's no way to enforce the strictly "every 10th" rule. You can do somethig like time based as @rnowitzki suggested or try to implement some INGEST_EVAL logic to base your decision on random value (and check if it falls within some range) or - for example - message hash. But this will of course give you some percentage on average, not a strict 1 out of every N policy.

0 Karma

rnowitzki
Builder

Hi @CMEOGNAD ,

See my response to a similiar question.  You could drop events with seconds *1 to *9, keep *0

It will not be exactly 9 of 10. But with something like that you could randomly reduce the volume. 

Not sure if it makes sense with your data.

(or Cribl)


--
Karma and/or Solution tagging appreciated.
0 Karma
Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...