Getting Data In
Highlighted

How do i exclude some events from being indexed by Splunk?

Splunk Employee
Splunk Employee

i have a data source that is very noisy, and i only want to index specific events from it, not all of them. for example, i only want to index logins and logouts, or login failures. how do i do this?

Tags (3)
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Splunk Employee
Splunk Employee

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

props.conf

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set = setnull, setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

View solution in original post

Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Splunk Employee
Splunk Employee

I prefer to use the regex (?=) to match anything. The regex . won't match if the field you're checking against happens to be empty, for example. Though this doesn't happen with the default _raw field, other fields can be empty.

Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Explorer

I gotta ask. What is that "5." in the code boxes. I am thinking it is a typo, but then again when it comes to configuration files, code, syntax and such one never know. I am trying to exclude some log entries with specific strings and it is not working yet, so then I think "well what is that 5. for" 🙂 🙂

0 Karma
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Splunk Employee
Splunk Employee

The 5 is a line count for the pasted data. Not relevant for actual usage in the config files.

0 Karma
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

New Member

I tried this solution but no success.
I am trying to filter data from being indexed.I need only the Error events

In props conf:
[source:://C:\Windows\System32\winevt\Logs]

Transforms must be applied in this order

to make sure events are dropped on the

floor prior to making their way to the

index processor

TRANSFORMS-set = setnull, setparsing

In transforms.conf:
[setnull]
REGEX = .
DESTKEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = Error
DEST
KEY = queue
FORMAT = indexQueue

0 Karma
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

New Member

I've also tried this but it doesn't work.

  • Could you clarify the location of the Props.conf and Transforms.conf files? There are several in the Splunk file system so it could be that I've modified the wrong ones?
0 Karma
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Path Finder

Correction: Works perfectly.

Issue is that I wasn't searching in the right time range.

0 Karma
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Splunk Employee
Splunk Employee

This example only includes things that contain 'login' and drops everthing else. Another use case would be to take in everything and make an exception for nosie you want filtered out.

The inverse to accept all except anything with the word 'info' would require just one stanza in transforms.conf:

[setnull]
REGEX = info
DEST_KEY = queue
FORMAT = nullQueue
Highlighted

Re: How do i exclude some events from being indexed by Splunk?

Splunk Employee
Splunk Employee

See this post:

http://answers.splunk.com/questions/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk

=========

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login

In props.conf:

[source::/var/log/foo]
# Transforms must be applied in this order
# to make sure events are dropped on the
# floor prior to making their way to the
# index processor
TRANSFORMS-set= setnull,setparsing

In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue