topic Re: How to filter a specific string from Splunk event logs at index-time? in Splunk Search

How to filter a specific string from Splunk event logs at index-time?

snehalk — Mon, 20 Jul 2015 10:57:46 GMT

Hello All,

I have requirement where need to filter(ignore) "---------------------------------------------" from the event logs, I have tried with the blacklist attribute in inputs.conf, but it is not working. Do we need to create props.conf and transforms.conf file, or we can ignore these dashes from inputs.conf only?

Please let me know which solution is best here, if we need to create a props.conf and transforms.conf, then what will be the contents of the files?

Thanks

Re: How to filter a specific string from Splunk event logs at index-time?

woodcock — Mon, 20 Jul 2015 16:41:42 GMT

Here is what you need to use the nullQueue:

http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues
http://answers.splunk.com/answers/86147/redirect-to-null-queue-is-not-working.html

Re: How to filter a specific string from Splunk event logs at index-time?

snehalk — Wed, 22 Jul 2015 14:42:13 GMT

Hello Woodcock,

Thanks for reply, as suggested I have deployed props.conf and transforms.conf on search heads but still the events are not filtering please let me know where i went wrong.

Took sourcetype instate of source

Props.conf

[sourcetype::mydata]
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = [^----------------------------------------]
DEST_KEY = queue
FORMAT = nullQueue

Re: How to filter a specific string from Splunk event logs at index-time?

woodcock — Wed, 29 Jul 2015 01:35:47 GMT

You need to deploy these to your Indexers (not your Search Head) and then restart the Splunk instances running on them. After that, the new data will be fixed but the old/existing data will still be "wrong". Also, use this (no square brackets) instead of what you have:

REGEX = ^----------------------------------------

Don't forget to "Accept" and answer to close the question.

Re: How to filter a specific string from Splunk event logs at index-time?

woodcock — Fri, 16 Oct 2015 16:35:38 GMT

Did this work?

Re: How to filter a specific string from Splunk event logs at index-time?

santiagoaloi — Fri, 16 Oct 2015 19:28:01 GMT

Hey!

First off these configurations happen in the parsing phase, before the indexing and searching phase, therefore
both props.conf and transforms.conf should be placed in the indexer, not the search head as you mentioned before. My suggestion is that you place these files in $SPLUNK_HOME/etc/system/local

The sourcetype stanza in props.conf isn't right, it should be as follows:

props.conf

 [mydata]
 TRANSFORMS-null = setnull

Your regex is not matching the repeated dashes in the event, so I put together a little regex for you.

transforms.conf

 [setnull]
 REGEX      = ^.*\s(\-+\S)
 DEST_KEY   = queue
 FORMAT     = nullQueue

Restart Splunk after changing these files.

If you don't wanna have the wrong old indexed data with the repeated dashes, one option is to use the DELETE command in a search query that matches those events with "------" in them. i.e:

index=yourindex sourcetype=mydata "---------------------------------------------" | DELETE

This will hide the matched events from further searches but will not erase them completely from the index.

Splunk by default doesn't allow users to run the command DELETE, so go to "Access controls » Roles » "
and apply the role "can_delete" to your user.

Hope this works for you 😃

/Santiago