Can you set up multiple filters for one log?

[agentguerry]

On your splunk indexers, can you set up multiple filters for one specific log?
I've been looking around, but maybe i'm not searching correctly.

ex.

/var/log/messages would have lines in it such as:

info: connected to remote host server.com
error: cannot ssh remote host server.com
critical: cannot ping remote host server.com

And say you want to filter out the lines that contain "info" and "error"

Can I have a "pre-indexing" filter to nullqueue any lines that have "info" or "error"?

Thanks.

[rsennett_splunk · Accepted solution]

Yes, You can have multiple transforms, and the context of your question (can I have multiple) you want individual TRANSFORMS- directives

In your case, to keep everything except events for info: and error: you would use two directives in props and match them in transforms.conf

This scenario is basically what the doc example shows: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad#Filter_event_data_a...

1. In props.conf:

   [source::/var/log/messages]
   

   TRANSFORMS-nullcrit = setnull-crit
   TRANSFORMS-nullerror = setnull-error

2. In transforms.conf:

   [setnull-error]
   REGEX = error:
   DEST_KEY = queue
   FORMAT = nullQueue

   [setnull-crit]
   REGEX = critical:
   DEST_KEY = queue
   FORMAT = nullQueue

[rsennett_splunk]

If this works for you, please come back and accept my answer. Thanks!

[agentguerry]

I guess I should specify.

I'd want to keep any message in /var/log/messages, but then not index "nullQ" error and critical

Should something like this be possible on the indexer?

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull-error,setnull-crit

transforms.conf
[setnull-error]
REGEX = error
DEST_KEY = queue
FORMAT = nullQueue

[setnull-crit]
REGEX = critical
DEST_KEY = queue
FORMAT = nullQueue

[rsennett_splunk]

I adjusted my answer to address the question as you show it. You need two separate TRANSFORMS directives on separate lines... and that'll do it.

[agentguerry]

I cannot wait to test this out! beautiful!

[rsennett_splunk]

awesome. 🙂
Sometimes when you first start messing with filtering... you might not see the results straight away for various reasons, all of them related to the act of testing. 🙂
A nice way to be sure you have no syntax errors, and see a nice satisfying result without having to restart the indexer, is to create a subset of sample data from your messages log.

add a stanza to props like this:
[nulltest]
TRANSFORMS-nullcrit = setnull-crit
TRANSFORMS-nullerror = setnullerror

and create the matching transforms.conf

Then, use the data onboarding wizard to just look at what the data will look like... if you pick your sourcetype (from the "custom" list) it will resolve it and you should see only the ones you are looking for (as long as the sample is deliberately short, of course) Super cool upgrade to the Onboarding wizard (formerly data previewer) - this way you don't have to pollute an index... since you don't have to really go all the way thru the wizard to see results.

Once you are sure... then put the props directives in the proper place and you'll have to restart the indexer to change the behavior on data that is already coming into Splunk...

[agentguerry]

I was under the assumption you have to restart splunk in order for the indexer to read its props/transforms files.

Is that not the case?

[rsennett_splunk]

only if you are making changes to an existing data source.

However, to test them... if you are going through the motions of on boarding something as "new", as with a sample file, the on boarding wizard will show you what it will look like if you did, go all the way to the index step. In fact, if you have a 'sandbox' index you could, go all the way and see the data indexed properly...
New data source, = no restart
Change to existing data source = restart

When you apply the changes to your REAL data, that is already being indexed... you have to restart for Splunk to apply the changes.