Drop a percentage of incoming events before hitting licensing processor for LnP use cases

[jethrop]

Hi guys.

We have a dev environment Splunk cluster with a dev license that LnP and dev teams send their data to.

They have a logging process on their systems, same as live, that is logging far too much data for our dev license.

They don't need the entire data set in dev,  30% for example is fine for their uses in development(not LnP) for testing dashboards etc.

To save them the need to re-write their code to only log every 3rd event, or a percentage of events for example, does anyone here know if it's possible to configure Splunk at input or Heavy Forwarder level to drop a percentage, or every x event for example?

[rnowitzki]

Hi @jethrop ,

If it can be pretty random which events are dropped, you could work with props and transforms on the HF to drop events based on the timestamp. For example drop events with the seconds 1*, 3*, 5* and keep all with 0*,2*,4*

You would just have to RegEx the seconds field and then follow the setup as documented here:

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

Never did that, but it should work.

Also it would be possible to do with Cribl, but adding this to the environment is maybe too much just for this use case.

BR
Ralph
--
Karma and/or Solution tagging appreciated.