topic Re: Event filter in Getting Data In

Event filter

jmallorquin — Wed, 27 Nov 2013 11:39:17 GMT

Hi to minimize the size of a index I would like to filter events for status 200

This is my config files:

inputs.conf
[monitor://C:\Logs*.log]
disabled = false
followTail = 0
host = CACA
index = basura
sourcetype = webexchange

props.conf

[webexchange]
TRANSFORMS-set= descartar

transforms.conf

[descartar]
REGEX = (?i)^(?:[^\.]*\.){8}\d+\s+(200)
DEST_KEY = queue
FORMAT = nullQueue

events:

2013-07-13 23:59:59 W3SVC1 222.222.222.222 HEAD /OAB/4abc7b21-fb88-473a-acfc-83660b79ff57/oab.xml - 443 - 172.26.12.166 Microsoft+BITS/7.5 401 2 2148074254

2013-07-14 00:00:00 W3SVC1 333.333.333.333 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Sync&User=xxxxxxxxxxxx&DeviceId=SEC1DCF083B0E526&DeviceType=SAMSUNGGTI9300&Log=V121_Fc1_Fid:8_Ty:Co_Filt0_St:S_Sk:1918965444_Sst9_LdapC0_LdapL0_RpcC21_RpcL31_Ers1_Pk3820395887_S1_ 443 xxxxxxx\xxxxxxxx 222.222.222.222 xxxxxxxxxxxxx/100.40102 200 0 0

2013-07-14 00:00:44 W3SVC1 333.333.333.333 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=xxxxxxxxxxxxx&DeviceId=SAMSUNG11091299461&DeviceType=SAMSUNGGTI9100&Log=V121_St:S_LdapC0_LdapL0_RpcC16_RpcL46_Pk3430192398_ 443 xxxxxxxx/xxxxxxx 222.222.2222.222 xxxxxxxxxxxxxxxxx/100.40102 200 0 0

Re: Event filter

martin_mueller — Wed, 27 Nov 2013 14:00:23 GMT

Do you have a question?

Re: Event filter

royimad — Wed, 27 Nov 2013 14:12:51 GMT

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login
In props.conf:

[source::/var/log/foo]

Transforms must be applied in this
order to make sure events are dropped
on the floor prior to making their way
to the index processor

TRANSFORMS-set= setnull,setparsing
In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

Re: Event filter

kristian_kolb — Wed, 27 Nov 2013 14:29:41 GMT

That seems like a very complicated (and possibly error-prone) regex for finding the events you want to filter out. Since the end of the message is much more predictable, it seems more convenient to anchor the regex there. Suggestion;

props.conf

[webexchange]
TRANSFORMS-set= descartar

transforms.conf

[descartar]
REGEX = \s200\s\d+\s\d+$
DEST_KEY = queue
FORMAT = nullQueue

Re: Event filter

jmallorquin — Wed, 27 Nov 2013 14:33:52 GMT

Thanks,

After all, the question is that with this configuration I can't filter events 200.

Re: Event filter

kristian_kolb — Wed, 27 Nov 2013 14:41:35 GMT

I did not see it as a question, either. And while your configuration (read: the regex) might work fine now, it is far safer to use the one I suggested below. Counting sequences of "non-dots followed by a dot" can break if there is an extra dot somewhere in an event, prior to the status code.

Re: Event filter

jmallorquin — Wed, 27 Nov 2013 14:46:21 GMT

Using my configuration and just changing the regex that you provided still doesn't work the filter with the events that I posted.

Re: Event filter

kristian_kolb — Wed, 27 Nov 2013 15:17:19 GMT

Are you making the configurations in the correct file/on the correct host?

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

Re: Event filter

lukejadamec — Wed, 27 Nov 2013 15:32:25 GMT

The filter will not affect events that have already been indexed, and the configs should be on the indexer or heavy forwarder that is doing the indexing. If you put the configs on the universal forwarder it will not work.

Re: Event filter

kristian_kolb — Wed, 27 Nov 2013 15:38:20 GMT

good point, forgot to mention the 'only affects new events' part.