topic Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file? in Splunk Search

How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

stanvv — Thu, 12 Nov 2015 11:52:18 GMT

Hi,

I only want to index files containing the string #! in the first 5 characters of the file.
Therefore, I created the following inputs.conf:

[monitor:pathname] 
blacklist = (?i:archive|develop|data|backup|\.txt$|\.gz$|\.tar$|\.csv$|\.bck$|\.log$|\.old$|\d{6,})
disabled = false 
host = script 
index = abcindex 
sourcetype = abcscript

Props.conf:

[abcscript] 
TRANSFORMS-set= setnull,setparsing

Transforms.conf:

[setnull] 
REGEX = . 
DEST_KEY = queue
FORMAT = nullQueue

[setparsing] 
REGEX = (.{0,5}(#!))
DEST_KEY = queue
FORMAT = indexQueue

Based on http://docs.splunk.com/Documentation/Splunk/6.3.1/Forwarding/Routeandfilterdatad
Unfortunately, everything is indexed in the index "abcindex" at the moment, and not only files starting with #!
I also tried it with a dummy string in a dummy file, but again, everything is indexed.
Rebooted Splunk after changing config files.

Any idea what goes wrong here?
Using Splunk 6.3.1 at the moment.

Thanks

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

woodcock — Thu, 12 Nov 2015 14:25:12 GMT

Your RegEx is wrong; try this:

REGEX = ^(.{0,3}(#!))

This needs to be deployed to all your indexers and the splunk instances running there need to be restarted. After this is done, incoming events will be properly filtered but events indexed before the restart will not be effected.

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

stanvv — Thu, 12 Nov 2015 14:41:03 GMT

Thanks for you answer. I tried the above (changed regex, rebooted and tried it with dummy files: one starting with #! and the other didn't) but still files not starting with #! were indexed.
I'm testing it on a local Splunk enterprise at the moment.

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

woodcock — Thu, 12 Nov 2015 17:35:23 GMT

The RegEx applies to each event, not to the entire file.

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

stanvv — Fri, 13 Nov 2015 11:46:14 GMT

The files I'm monitoring are scripts (sometimes with an undefined filetype). So if the file content itself starts with #! I want it to be indexed. If it doesn't, it should go to the nullQueue.

Example
File 1 (needs to be indexed)

#!
########
# Intro 123
########
#Scriptinfo
ABC = 123

File 2 (send to nullQueue)

########
# Intro 234
########
#Scriptinfo
DEF = 567

Do props.conf and transforms.conf also work for non log/txt files? Any ideas what's the best solution?

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

tmarlette — Mon, 16 Nov 2015 19:28:42 GMT

Out of curiosity are you trying to do all of this on a universal forwarder?

If you are, adding these props/transforms to a UF they won't work, you have to add those settings to your indexing tier.

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

stanvv — Wed, 18 Nov 2015 10:29:55 GMT

I'm testing it on a local Splunk enterprise at the moment.

Re: How to filter out events in Splunk 6.3.1 at index-time, except files containing the string "#!" in the first 5 characters of the file?

woodcock — Wed, 18 Nov 2015 23:05:41 GMT

Make sure you set this for your sourcetype in props.conf:

[YourSourcetypeHere]
LINE_BREAKER=(\Z)
TRUNCATE=500000
SHOULD_LINEMERGE = 1

This will treat the entire file as a single event and then it should work as you expect. Deploy this to the Indexers (or Heavy Forwarders) and restart all splunk instances there. This will apply ONLY TO FUTURE EVENTS (the scripts that are already there have already been processed) so you will have to create new files in order to test this.