topic Re: How do i exclude some events from being indexed by Splunk? in Getting Data In

How do i exclude some events from being indexed by Splunk?

piebob — Fri, 15 Jan 2010 07:45:28 GMT

i have a data source that is very noisy, and i only want to index specific events from it, not all of them. for example, i only want to index logins and logouts, or login failures. how do i do this?

Re: How do i exclude some events from being indexed by Splunk?

matt — Fri, 15 Jan 2010 08:01:09 GMT

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

props.conf

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set = setnull, setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

Re: How do i exclude some events from being indexed by Splunk?

gkanapathy — Fri, 15 Jan 2010 09:10:11 GMT

I prefer to use the regex (?=) to match anything. The regex . won't match if the field you're checking against happens to be empty, for example. Though this doesn't happen with the default _raw field, other fields can be empty.

Re: How do i exclude some events from being indexed by Splunk?

dskillman — Fri, 09 Apr 2010 02:56:36 GMT

This example only includes things that contain 'login' and drops everthing else. Another use case would be to take in everything and make an exception for nosie you want filtered out.

The inverse to accept all except anything with the word 'info' would require just one stanza in transforms.conf:

[setnull]
REGEX = info
DEST_KEY = queue
FORMAT = nullQueue

Re: How do i exclude some events from being indexed by Splunk?

Simeon — Fri, 30 Apr 2010 05:56:25 GMT

See this post:

http://answers.splunk.com/questions/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk

=========

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

In props.conf:

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

Re: How do i exclude some events from being indexed by Splunk?

geoeldsul — Wed, 04 May 2016 13:15:18 GMT

I gotta ask. What is that "5." in the code boxes. I am thinking it is a typo, but then again when it comes to configuration files, code, syntax and such one never know. I am trying to exclude some log entries with specific strings and it is not working yet, so then I think "well what is that 5. for" 🙂 🙂

Re: How do i exclude some events from being indexed by Splunk?

esix_splunk — Mon, 09 May 2016 02:50:01 GMT

The 5 is a line count for the pasted data. Not relevant for actual usage in the config files.

Re: How do i exclude some events from being indexed by Splunk?

reswob4 — Wed, 16 Aug 2017 14:26:44 GMT

I downvoted this post because recursive. the link under "see this post" points back to this question.

Re: How do i exclude some events from being indexed by Splunk?

amit2301 — Tue, 29 Sep 2020 15:52:14 GMT

I tried this solution but no success.
I am trying to filter data from being indexed.I need only the Error events

In props conf:
[source:://C:\Windows\System32\winevt\Logs]

Transforms must be applied in this order

to make sure events are dropped on the

floor prior to making their way to the

index processor

TRANSFORMS-set = setnull, setparsing

In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = Error
DEST_KEY = queue
FORMAT = indexQueue

Re: How do i exclude some events from being indexed by Splunk?

kalpeshkhetanil — Wed, 31 Oct 2018 09:55:43 GMT

I've also tried this but it doesn't work.

Could you clarify the location of the Props.conf and Transforms.conf files? There are several in the Splunk file system so it could be that I've modified the wrong ones?

Re: How do i exclude some events from being indexed by Splunk?

ifeldshteyn — Wed, 22 Jan 2020 17:17:55 GMT

Correction: Works perfectly.

Issue is that I wasn't searching in the right time range.