topic Re: how to throttle some data from being indexed in Getting Data In

how to throttle some data from being indexed

HadvoraMaya — Mon, 26 Feb 2018 12:17:25 GMT

Hi,

I have an event that is a real license consumer. I would like to throttle only this event. I want only 1 of 10 hits of the same event will be indexed. All other events will remain the same with no change.

How can I do that?

Thanks,
Michael

Re: how to throttle some data from being indexed

FrankVl — Mon, 26 Feb 2018 12:49:07 GMT

How are you currently ingesting that data? On a UF or a HF, through what input method?

Re: how to throttle some data from being indexed

HadvoraMaya — Mon, 26 Feb 2018 13:17:49 GMT

I "Shoot" the data from the Application into Splunk engine via a specific port.
Not using Splunk Forwarder.

Re: how to throttle some data from being indexed

FrankVl — Mon, 26 Feb 2018 15:18:26 GMT

You can drop events matching a certain regex by assigning them to the null queue:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

Question is how you're going to decide which 1 event to keep and which 9 events to drop. If they are somewhat uniformly distributed across time, you could perhaps say everything with milisecond 1-9 goes to nullqueue and milisecond 0 goes to parsing queue or something, but that's rather tricky. But as you know the data, perhaps you can think of a field that provides some uniform distribution that you could use for making a 10%-90% split...

Re: how to throttle some data from being indexed

HadvoraMaya — Mon, 26 Feb 2018 15:28:41 GMT

I know the event's name. I just want it to filter out 90% of this event's beeing indexed.
Just need to be able to say that event name MM should be index 1 out of 10 events.

Re: how to throttle some data from being indexed

FrankVl — Tue, 27 Feb 2018 08:59:08 GMT

And as I mentioned: the only thing I can think of to do that is find a regex that (probably more or randomly) matches 10% of the events. E.g. by triggering of the milliseconds or maybe there is some incremental eventID where you could ignore all eventIDs that end with 1-9 and only accept eventIDs ending in 0 or something.

As mentioned: this is all not brilliantly reliable, but it's the best I can think of. As far as I know there is no way to tell splunk to let 1 out 10 events through.