Installation
Highlighted

Filter Indexing to Avoid License Issues

Contributor

i have Splunk monitoring some syslog files. i also have lots of noisy syslog going into syslog, much of it is noise i dont care to index. with just 11 devices i am at 80MB/hr but want to limit that by adding filters to Splunk to not index certain syslog message types, like tcp teardown from my ASA firewalls, etc.

while posting this i did see some suggested related topics but the Q's were either from 2010, or were more recent but relevant to a older version of Splunk.

is there a easy way to click-to-filter events so they are no longer indexed, etc ?

Labels (1)
Highlighted

Re: Filter Indexing to Avoid License Issues

Splunk Employee
Splunk Employee

To answer your question in a word, No. You can't click to filter events. However, you can route data you don't want indexed into the nullQueue so you won't incur the licensing cost indexing those events would carry.

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad

Specifically, you should review the topic 'Discard specific events and keep the rest'. It gives an example of doing exactly what you're inquiring about in your question.

View solution in original post

Highlighted

Re: Filter Indexing to Avoid License Issues

Contributor

jbsplunk,

the documentation says:

  1. In props.conf, set the TRANSFORMS-null attribute:
    [source::/var/log/messages]
    TRANSFORMS-null= setnull

  2. Create a corresponding stanza in transforms.conf. Set DESTKEY to "queue" and FORMAT to "nullQueue":
    [setnull]
    REGEX = \[sshd\]
    DEST
    KEY = queue
    FORMAT = nullQueue

i have 11 log files (from syslog) being monitored, each in their own dir (and more coming), so can i use wildcard for source like [source::/syslog/*/*/*/*/log] ? and then use OR in REGEX like \[sshd\]|TEARDOWN|FOO ? or do i need to create conf statements for each log file?

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Splunk Employee
Splunk Employee

use OR in REGEX like [sshd]|TEARDOWN|FOO is what you're looking for, as Splunk supports PCRE. If you can do it in PCRE, you can do it in Splunk.

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Contributor

comments need \ escaping if you wish to use the \ char in the comment, notice your sshd text is missing the escape char. my ? char is not part of the regex, just there as questionmark punctuation, etc. i tried exactly what i posted but its not working, i suspect the source using wildcard * is not valid ??

this is my actual transdforms syntax:
[setnull]
REGEX = ASA-6-(302021|302020|305011|305013|305014|305012)
DEST_KEY = queue
FORMAT = nullQueue

i tried using actual path for source in props.conf but the events are still showing up. any suggestions?

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Splunk Employee
Splunk Employee

Yeah, understood. I had just copied/pasted, and must have missed a character. Anyway, that looks like a transform I've set up in the past for someone else, so I think it will work well for you. If you found the information helpful, please feel free to upvote and accept the answer.

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Contributor

ok, but its not working. i still get the events in a search. i did restart Splunk after the edits, etc. i am running latest rev, etc.

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Splunk Employee
Splunk Employee

What do your props look like? I don't know how you're calling your transforms.conf, so it could be that you're doing that incorrectly. When this isn't working, I try to make it as simple as possible. For instance, try calling it using sourcetype, and for your regex, use '.', so everything should get routed to the nullQueue. That way, at least I can know if the problem is with the transform or someplace in props.

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Contributor

all i did was add to the conf files, the documentation says to add to the conf files, and "thats it".

transforms.conf
[setnull]
REGEX = ASA-6-(302021|302020|302014|305011|305013|305014|305012)
DEST_KEY = queue
FORMAT = nullQueue

and props.conf
[source::/logs/syslog/firewalls/fw1/2010/03/log]
TRANSFORMS-null= setnull

how do you mean "call" the transform?

EDIT - i just noticed source error, should be 2012 not 2010. edited and restarting now, so lets see, 1min.

0 Karma
Highlighted

Re: Filter Indexing to Avoid License Issues

Splunk Employee
Splunk Employee

I mean by calling the transform in props, as you've done. What is the source field that is shown in your UI? Does it match what you've got here? If not, it probably won't work. That's why it might be easier to use the sourcetype field instead.

0 Karma