need to route data to nullqueue based on index

[purnavenkatesh]

Hi,

I need to route the index data to null-queue based on the strings from the events. For example, all the events that contain string pattern "Error" from all the QA* indexes should to routed to nullqueue.

Doing it with host and sourcetype is very complex. Can someone suggest me with solution?

[kmugglet · Accepted solution]

Hi @purnavenkatesh,

Basically this splunk doc answers your questions.

You'll be matching events based on source, sourcetype, host or index in your props.conf and running a transform on that event to change it's destination index to nullQueue.

[purnavenkatesh]

I tried the props stanza with index and it didn't work. Looking for other approach to achieve this.

[purnavenkatesh]

doing this using host stanza.

[FrankVl]

Wouldn't it be possible to follow those routing and filtering instructions and write a transforms stanza that uses SOURCE_KEY = _MetaData:Index, to target events for a certain index?

So write 1 transforms that assigns everything that matches ERROR to the nullqueue, and the overrule that for events that have Index=PROD? Or something along those lines?

[xpac]

I thought about something similar.
Possible, but really dirty hack:

• For all events, append the content of index to _raw, using a very creative separator. E.g. an event yourtext becomes yourtext#&#&#&QA_main
• For all events, match on a regex like .*ERROR.*#&#&#&QA.* and route all events to nullQueue
• For all events, remove the append separator and index again.

That should work, because you could match ERROR and index in the same step, however it would require three steps to be performed on ALL events, which might get ressource heavy and also is dirty like hell.

[MuS]

little correction here, props.conf stanzas cannot be applied to indexes - see the docs on props.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

[<spec>]
* This stanza enables properties for a given <spec>.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host, or host-matching pattern, for an event.
3. source::<source>, where <source> is the source, or source-matching pattern, for an event.

[arunsunny]

Can we use a wildcard something like below?

[source::*]
[host::]
[]

[xpac]

Just checking - you want to drop certain events that are supposed to go to a certain index (or some indexes), and also contain a certain string?

[purnavenkatesh]

yes, to be more clear.

I want to drop all the events with string ERROR in it from set of indexes whose starting string is QA

[xpac]

I fear that's not possible at index time, because you can only filter on either the event text OR the index name. It would be much easier if those events have a common set of source, sourcetype or host, then filtering on the event text would be easy.

[purnavenkatesh]

For sourcetype: though they have the common sourcetype, PROD index is also using the same sourcetype. Hence, dropping the data using soucetype will drop the prod data with matching pattern. Hence this is ruled out.

For host: the combination are too many and it is going very complex.

[xpac]

Sorry, I don't have any good ideas on this that don't sound like really dirty hacks. I'd try to somehow get this organized with hosts and sourcetypes, but I can see how this can be difficult on it's own.