Splunk Search

need to route data to nullqueue based on index

purnavenkatesh
Explorer

Hi,

I need to route the index data to null-queue based on the strings from the events. For example, all the events that contain string pattern "Error" from all the QA* indexes should to routed to nullqueue.

Doing it with host and sourcetype is very complex. Can someone suggest me with solution?

Tags (1)
0 Karma
1 Solution

kmugglet
Communicator

Hi @purnavenkatesh,

Basically this splunk doc answers your questions.

You'll be matching events based on source, sourcetype, host or index in your props.conf and running a transform on that event to change it's destination index to nullQueue.

View solution in original post

0 Karma

kmugglet
Communicator

Hi @purnavenkatesh,

Basically this splunk doc answers your questions.

You'll be matching events based on source, sourcetype, host or index in your props.conf and running a transform on that event to change it's destination index to nullQueue.

0 Karma

purnavenkatesh
Explorer

I tried the props stanza with index and it didn't work. Looking for other approach to achieve this.

0 Karma

purnavenkatesh
Explorer

doing this using host stanza.

0 Karma

FrankVl
Ultra Champion

Wouldn't it be possible to follow those routing and filtering instructions and write a transforms stanza that uses SOURCE_KEY = _MetaData:Index, to target events for a certain index?

So write 1 transforms that assigns everything that matches ERROR to the nullqueue, and the overrule that for events that have Index=PROD? Or something along those lines?

0 Karma

xpac
SplunkTrust
SplunkTrust

I thought about something similar.
Possible, but really dirty hack:

  • For all events, append the content of index to _raw, using a very creative separator. E.g. an event yourtext becomes yourtext#&#&#&QA_main
  • For all events, match on a regex like .*ERROR.*#&#&#&QA.* and route all events to nullQueue
  • For all events, remove the append separator and index again.

That should work, because you could match ERROR and index in the same step, however it would require three steps to be performed on ALL events, which might get ressource heavy and also is dirty like hell.

0 Karma

MuS
Legend

little correction here, props.conf stanzas cannot be applied to indexes - see the docs on props.conf http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

[<spec>]
* This stanza enables properties for a given <spec>.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host, or host-matching pattern, for an event.
3. source::<source>, where <source> is the source, or source-matching pattern, for an event.

arunsunny
Path Finder

Can we use a wildcard something like below?

[source::*]
[host::]
[
]

0 Karma

xpac
SplunkTrust
SplunkTrust

Just checking - you want to drop certain events that are supposed to go to a certain index (or some indexes), and also contain a certain string?

0 Karma

purnavenkatesh
Explorer

yes, to be more clear.

I want to drop all the events with string ERROR in it from set of indexes whose starting string is QA

0 Karma

xpac
SplunkTrust
SplunkTrust

I fear that's not possible at index time, because you can only filter on either the event text OR the index name. It would be much easier if those events have a common set of source, sourcetype or host, then filtering on the event text would be easy.

0 Karma

purnavenkatesh
Explorer

For sourcetype: though they have the common sourcetype, PROD index is also using the same sourcetype. Hence, dropping the data using soucetype will drop the prod data with matching pattern. Hence this is ruled out.

For host: the combination are too many and it is going very complex.

0 Karma

xpac
SplunkTrust
SplunkTrust

Sorry, I don't have any good ideas on this that don't sound like really dirty hacks. I'd try to somehow get this organized with hosts and sourcetypes, but I can see how this can be difficult on it's own.

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...