Getting Data In

Drop a percentage of incoming events before hitting licensing processor for LnP use cases

jethrop
Explorer

Hi guys.

We have a dev environment Splunk cluster with a dev license that LnP and dev teams send their data to.

They have a logging process on their systems, same as live, that is logging far too much data for our dev license.

They don't need the entire data set in dev,  30% for example is fine for their uses in development(not LnP) for testing dashboards etc.

To save them the need to re-write their code to only log every 3rd event, or a percentage of events for example, does anyone here know if it's possible to configure Splunk at input or Heavy Forwarder level to drop a percentage, or every x event for example?

0 Karma

rnowitzki
Builder

Hi @jethrop ,

If it can be pretty random which events are dropped, you could work with props and transforms on the HF to drop events based on the timestamp. For example drop events with the seconds 1*, 3*, 5* and keep all with 0*,2*,4*

You would just have to RegEx the seconds field and then follow the setup as documented here:

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

Never did that, but it should work.

Also it would be possible to do with Cribl, but adding this to the environment is maybe too much just for this use case.


BR
Ralph
--
Karma and/or Solution tagging appreciated.

--
Karma and/or Solution tagging appreciated.
0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...