Getting Data In

Can we do the event sampling before indexing the data?

kml_uvce
Builder

Can we do the event sampling in forwarder before indexing the event in indexer to reduce the event size ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, you can.  Splunk 9.0 introduces the Ingest Actions feature that allows indexers and heavy forwarders to filter data.  See https://docs.splunk.com/Documentation/Splunk/9.0.0/ReleaseNotes/MeetSplunk#What.27s_New_in_9.0 for more.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kml_uvce
Builder

Thanks @richgalloway : can we do the sampling in forwarder in splunk versions 7.2  or old version

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Heavy forwarders and indexers older than 9.0 can do event filtering using props and transforms.  In the transform, a regular expression is matched against incoming events.  Events that match the regex are sent to the null queue (discarded).

No Universal Forwarder can filter events.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kml_uvce
Builder

@richgalloway 

I am looking for sampling like 10:1 or 100:1 ratio in events to filter out events , not looking for simple filter that we can do in heavy forwarder

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Since every event is parsed and indexed separately, you can't "count" the events to choose only each 10th or 100th event. But you can try to calculate some value based on a message that will match given value roughly 1/n times in the long run. You could, for example, calculate hash from an event and check its last one or two digits.

But this will not give you perfect sampling ratio. And with badly chosen function your results may be skewed.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...