How do i exclude some events from being indexed by Splunk?

[piebob]

i have a data source that is very noisy, and i only want to index specific events from it, not all of them. for example, i only want to index logins and logouts, or login failures. how do i do this?

[Simeon]

See this post:

http://answers.splunk.com/questions/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk

=========

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

In props.conf:

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

[reswob4]

I downvoted this post because recursive. the link under "see this post" points back to this question.

[dskillman]

This example only includes things that contain 'login' and drops everthing else. Another use case would be to take in everything and make an exception for nosie you want filtered out.

The inverse to accept all except anything with the word 'info' would require just one stanza in transforms.conf:

[setnull]
REGEX = info
DEST_KEY = queue
FORMAT = nullQueue

[matt · Accepted solution]

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

props.conf

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set = setnull, setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

[amit2301]

I tried this solution but no success.
I am trying to filter data from being indexed.I need only the Error events

In props conf:
[source:://C:\Windows\System32\winevt\Logs]

Transforms must be applied in this order

to make sure events are dropped on the

floor prior to making their way to the

index processor

TRANSFORMS-set = setnull, setparsing

In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = Error
DEST_KEY = queue
FORMAT = indexQueue

[kalpeshkhetanil]

I've also tried this but it doesn't work.

• Could you clarify the location of the Props.conf and Transforms.conf files? There are several in the Splunk file system so it could be that I've modified the wrong ones?

[ifeldshteyn]

Correction: Works perfectly.

Issue is that I wasn't searching in the right time range.

[geoeldsul]

I gotta ask. What is that "5." in the code boxes. I am thinking it is a typo, but then again when it comes to configuration files, code, syntax and such one never know. I am trying to exclude some log entries with specific strings and it is not working yet, so then I think "well what is that 5. for" 🙂 🙂

[esix_splunk]

The 5 is a line count for the pasted data. Not relevant for actual usage in the config files.

[gkanapathy]

I prefer to use the regex (?=) to match anything. The regex . won't match if the field you're checking against happens to be empty, for example. Though this doesn't happen with the default _raw field, other fields can be empty.