Getting Data In

Drop a percentage of incoming events before hitting licensing processor for LnP use cases

jethrop
Explorer

Hi guys.

We have a dev environment Splunk cluster with a dev license that LnP and dev teams send their data to.

They have a logging process on their systems, same as live, that is logging far too much data for our dev license.

They don't need the entire data set in dev,  30% for example is fine for their uses in development(not LnP) for testing dashboards etc.

To save them the need to re-write their code to only log every 3rd event, or a percentage of events for example, does anyone here know if it's possible to configure Splunk at input or Heavy Forwarder level to drop a percentage, or every x event for example?

0 Karma

rnowitzki
Builder

Hi @jethrop ,

If it can be pretty random which events are dropped, you could work with props and transforms on the HF to drop events based on the timestamp. For example drop events with the seconds 1*, 3*, 5* and keep all with 0*,2*,4*

You would just have to RegEx the seconds field and then follow the setup as documented here:

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

Never did that, but it should work.

Also it would be possible to do with Cribl, but adding this to the environment is maybe too much just for this use case.


BR
Ralph
--
Karma and/or Solution tagging appreciated.

--
Karma and/or Solution tagging appreciated.
0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!