topic Re: Filter data out from Data Input in Getting Data In

Filter data out from Data Input

bahmed — Wed, 16 Apr 2014 20:46:13 GMT

Hi,

I have spent last 2 hours searching for this simple scenario on Splunk Answers, without any luck.
Here is the case.

Splunk 6.0.2 (Trial version)
OS : Windows 7, 64 Bit
Data Input : A Log4J file on my local computer

Requirement : Just want to index events which contains the string "[ERROR ]", in my indexer.

Any help will be greatly appreciated.

Re: Filter data out from Data Input

yannK — Wed, 16 Apr 2014 21:24:28 GMT

Setup those type of filters at the indexer level :
You can use a rule based on the sourcetype, and a matching regex based on the event.
You can test your regex on sample events with the "rex" command in splunk before to make sure.

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Re: Filter data out from Data Input

bahmed — Tue, 22 Apr 2014 20:59:05 GMT

Based on the article, I set the following files as shown, but still not getting the filtered log.

props.config

[MyVacationLog]
TRANSFORMS-set= myvacnull,myvacparsing

transforms.config

[myvacnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[myvacparsing]
REGEX = [ERROR]
DEST_KEY = queue
FORMAT = indexQueue

Re: Filter data out from Data Input

yannK — Tue, 22 Apr 2014 21:17:47 GMT

1- [ and ] are regex keyword, you should escape them
REGEX = \[ERROR\]

2- make sure that those props/transforms are on the indexers (not on universal or lightweight forwarders only)

Re: Filter data out from Data Input

bahmed — Wed, 23 Apr 2014 16:45:31 GMT

Thanks yannK. I have changed the regular expression as per your point 1. No change in the result.

My props/transforms are in the following directory : C:\Program Files\Splunk\etc\system\local

Does that makes them on indexers.

Appreciate your help.

Re: Filter data out from Data Input

yannK — Wed, 23 Apr 2014 19:00:36 GMT

This looked correct, What is the current behavior when you index new events :

all events are dropped the nullQueue (with or without the [ERROR] keyword)
all events are indexed ?
a mix of both ?

did you restarted the indexers to apply the change ?

Re: Filter data out from Data Input

bahmed — Wed, 23 Apr 2014 21:05:46 GMT

For the new events, all of them are getting indexed including the one contains "Error".

I have restarted the Splunk on my local machine.