topic Re: need to route data to nullqueue based on index in Splunk Search

need to route data to nullqueue based on index

purnavenkatesh — Wed, 02 May 2018 21:40:32 GMT

Hi,

I need to route the index data to null-queue based on the strings from the events. For example, all the events that contain string pattern "Error" from all the QA* indexes should to routed to nullqueue.

Doing it with host and sourcetype is very complex. Can someone suggest me with solution?

Re: need to route data to nullqueue based on index

xpac — Wed, 02 May 2018 21:58:05 GMT

Just checking - you want to drop certain events that are supposed to go to a certain index (or some indexes), and also contain a certain string?

Re: need to route data to nullqueue based on index

purnavenkatesh — Wed, 02 May 2018 22:15:05 GMT

yes, to be more clear.

I want to drop all the events with string ERROR in it from set of indexes whose starting string is QA

Re: need to route data to nullqueue based on index

xpac — Wed, 02 May 2018 22:28:58 GMT

I fear that's not possible at index time, because you can only filter on either the event text OR the index name. It would be much easier if those events have a common set of source, sourcetype or host, then filtering on the event text would be easy.

Re: need to route data to nullqueue based on index

purnavenkatesh — Wed, 02 May 2018 22:45:13 GMT

For sourcetype: though they have the common sourcetype, PROD index is also using the same sourcetype. Hence, dropping the data using soucetype will drop the prod data with matching pattern. Hence this is ruled out.

For host: the combination are too many and it is going very complex.

Re: need to route data to nullqueue based on index

xpac — Thu, 03 May 2018 00:29:42 GMT

Sorry, I don't have any good ideas on this that don't sound like really dirty hacks. I'd try to somehow get this organized with hosts and sourcetypes, but I can see how this can be difficult on it's own.

Re: need to route data to nullqueue based on index

kmugglet — Thu, 03 May 2018 01:11:10 GMT

Hi @purnavenkatesh,

Basically this splunk doc answers your questions.

You'll be matching events based on source, sourcetype, host or index in your props.conf and running a transform on that event to change it's destination index to nullQueue.

Re: need to route data to nullqueue based on index

MuS — Thu, 03 May 2018 02:10:17 GMT

little correction here, props.conf stanzas cannot be applied to indexes - see the docs on props.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

[<spec>]
* This stanza enables properties for a given <spec>.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host, or host-matching pattern, for an event.
3. source::<source>, where <source> is the source, or source-matching pattern, for an event.

Re: need to route data to nullqueue based on index

purnavenkatesh — Thu, 03 May 2018 02:28:06 GMT

I tried the props stanza with index and it didn't work. Looking for other approach to achieve this.

Re: need to route data to nullqueue based on index

FrankVl — Thu, 03 May 2018 07:19:50 GMT

Wouldn't it be possible to follow those routing and filtering instructions and write a transforms stanza that uses SOURCE_KEY = _MetaData:Index, to target events for a certain index?

So write 1 transforms that assigns everything that matches ERROR to the nullqueue, and the overrule that for events that have Index=PROD? Or something along those lines?

Re: need to route data to nullqueue based on index

xpac — Tue, 29 Sep 2020 19:20:31 GMT

I thought about something similar.
Possible, but really dirty hack:

For all events, append the content of index to _raw, using a very creative separator. E.g. an event yourtext becomes yourtext#&#&#&QA_main
For all events, match on a regex like .*ERROR.*#&#&#&QA.* and route all events to nullQueue
For all events, remove the append separator and index again.

That should work, because you could match ERROR and index in the same step, however it would require three steps to be performed on ALL events, which might get ressource heavy and also is dirty like hell.

Re: need to route data to nullqueue based on index

purnavenkatesh — Tue, 08 May 2018 22:08:54 GMT

doing this using host stanza.

Re: need to route data to nullqueue based on index

arunsunny — Thu, 26 Sep 2019 04:29:33 GMT

Can we use a wildcard something like below?

[source::*]
[host::]
[]