Getting Data In

Can you set up multiple filters for one log?

agentguerry
Explorer

On your splunk indexers, can you set up multiple filters for one specific log?
I've been looking around, but maybe i'm not searching correctly.

ex.

/var/log/messages would have lines in it such as:

info: connected to remote host server.com
error: cannot ssh remote host server.com
critical: cannot ping remote host server.com

And say you want to filter out the lines that contain "info" and "error"

Can I have a "pre-indexing" filter to nullqueue any lines that have "info" or "error"?

Thanks.

0 Karma
1 Solution

rsennett_splunk
Splunk Employee
Splunk Employee

Yes, You can have multiple transforms, and the context of your question (can I have multiple) you want individual TRANSFORMS- directives

In your case, to keep everything except events for info: and error: you would use two directives in props and match them in transforms.conf

This scenario is basically what the doc example shows: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad#Filter_event_data_a...

  1. In props.conf:

    [source::/var/log/messages]
    

    TRANSFORMS-nullcrit = setnull-crit
    TRANSFORMS-nullerror = setnull-error

  2. In transforms.conf:

    [setnull-error]
    REGEX = error:
    DEST_KEY = queue
    FORMAT = nullQueue

    [setnull-crit]
    REGEX = critical:
    DEST_KEY = queue
    FORMAT = nullQueue

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

rsennett_splunk
Splunk Employee
Splunk Employee

Yes, You can have multiple transforms, and the context of your question (can I have multiple) you want individual TRANSFORMS- directives

In your case, to keep everything except events for info: and error: you would use two directives in props and match them in transforms.conf

This scenario is basically what the doc example shows: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad#Filter_event_data_a...

  1. In props.conf:

    [source::/var/log/messages]
    

    TRANSFORMS-nullcrit = setnull-crit
    TRANSFORMS-nullerror = setnull-error

  2. In transforms.conf:

    [setnull-error]
    REGEX = error:
    DEST_KEY = queue
    FORMAT = nullQueue

    [setnull-crit]
    REGEX = critical:
    DEST_KEY = queue
    FORMAT = nullQueue

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

rsennett_splunk
Splunk Employee
Splunk Employee

If this works for you, please come back and accept my answer. Thanks!

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agentguerry
Explorer

I guess I should specify.

I'd want to keep any message in /var/log/messages, but then not index "nullQ" error and critical

Should something like this be possible on the indexer?

props.conf
[source::/var/log/messages]
TRANSFORMS-set= setnull-error,setnull-crit

transforms.conf
[setnull-error]
REGEX = error
DEST_KEY = queue
FORMAT = nullQueue

[setnull-crit]
REGEX = critical
DEST_KEY = queue
FORMAT = nullQueue

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

I adjusted my answer to address the question as you show it. You need two separate TRANSFORMS directives on separate lines... and that'll do it.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agentguerry
Explorer

I cannot wait to test this out! beautiful!

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

awesome. 🙂
Sometimes when you first start messing with filtering... you might not see the results straight away for various reasons, all of them related to the act of testing. 🙂
A nice way to be sure you have no syntax errors, and see a nice satisfying result without having to restart the indexer, is to create a subset of sample data from your messages log.

add a stanza to props like this:
[nulltest]
TRANSFORMS-nullcrit = setnull-crit
TRANSFORMS-nullerror = setnullerror

and create the matching transforms.conf

Then, use the data onboarding wizard to just look at what the data will look like... if you pick your sourcetype (from the "custom" list) it will resolve it and you should see only the ones you are looking for (as long as the sample is deliberately short, of course) Super cool upgrade to the Onboarding wizard (formerly data previewer) - this way you don't have to pollute an index... since you don't have to really go all the way thru the wizard to see results.

Once you are sure... then put the props directives in the proper place and you'll have to restart the indexer to change the behavior on data that is already coming into Splunk...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

agentguerry
Explorer

I was under the assumption you have to restart splunk in order for the indexer to read its props/transforms files.

Is that not the case?

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

only if you are making changes to an existing data source.

However, to test them... if you are going through the motions of on boarding something as "new", as with a sample file, the on boarding wizard will show you what it will look like if you did, go all the way to the index step. In fact, if you have a 'sandbox' index you could, go all the way and see the data indexed properly...
New data source, = no restart
Change to existing data source = restart

When you apply the changes to your REAL data, that is already being indexed... you have to restart for Splunk to apply the changes.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.