Event filter

[jmallorquin]

Hi to minimize the size of a index I would like to filter events for status 200

This is my config files:

inputs.conf
[monitor://C:\Logs*.log]
disabled = false
followTail = 0
host = CACA
index = basura
sourcetype = webexchange

props.conf

[webexchange]
TRANSFORMS-set= descartar

transforms.conf

[descartar]
REGEX = (?i)^(?:[^\.]*\.){8}\d+\s+(200)
DEST_KEY = queue
FORMAT = nullQueue

events:

2013-07-13 23:59:59 W3SVC1 222.222.222.222 HEAD /OAB/4abc7b21-fb88-473a-acfc-83660b79ff57/oab.xml - 443 - 172.26.12.166 Microsoft+BITS/7.5 401 2 2148074254

2013-07-14 00:00:00 W3SVC1 333.333.333.333 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Sync&User=xxxxxxxxxxxx&DeviceId=SEC1DCF083B0E526&DeviceType=SAMSUNGGTI9300&Log=V121_Fc1_Fid:8_Ty:Co_Filt0_St:S_Sk:1918965444_Sst9_LdapC0_LdapL0_RpcC21_RpcL31_Ers1_Pk3820395887_S1_ 443 xxxxxxx\xxxxxxxx 222.222.222.222 xxxxxxxxxxxxx/100.40102 200 0 0

2013-07-14 00:00:44 W3SVC1 333.333.333.333 POST /Microsoft-Server-ActiveSync/default.eas Cmd=FolderSync&User=xxxxxxxxxxxxx&DeviceId=SAMSUNG11091299461&DeviceType=SAMSUNGGTI9100&Log=V121_St:S_LdapC0_LdapL0_RpcC16_RpcL46_Pk3430192398_ 443 xxxxxxxx/xxxxxxx 222.222.2222.222 xxxxxxxxxxxxxxxxx/100.40102 200 0 0

[kristian_kolb]

good point, forgot to mention the 'only affects new events' part.

[lukejadamec]

The filter will not affect events that have already been indexed, and the configs should be on the indexer or heavy forwarder that is doing the indexing. If you put the configs on the universal forwarder it will not work.

[kristian_kolb]

Are you making the configurations in the correct file/on the correct host?

http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

[jmallorquin]

Using my configuration and just changing the regex that you provided still doesn't work the filter with the events that I posted.

[kristian_kolb]

I did not see it as a question, either. And while your configuration (read: the regex) might work fine now, it is far safer to use the one I suggested below. Counting sequences of "non-dots followed by a dot" can break if there is an extra dot somewhere in an event, prior to the status code.

/k

[jmallorquin]

Thanks,

After all, the question is that with this configuration I can't filter events 200.

[kristian_kolb]

That seems like a very complicated (and possibly error-prone) regex for finding the events you want to filter out. Since the end of the message is much more predictable, it seems more convenient to anchor the regex there. Suggestion;

props.conf

[webexchange]
TRANSFORMS-set= descartar

transforms.conf

[descartar]
REGEX = \s200\s\d+\s\d+$
DEST_KEY = queue
FORMAT = nullQueue

/k

[royimad]

This is done by defining a regex to match the necessary event(s) and send everything else to nullqueue

Here is a basic example that will drop everything except events that contain the string login
In props.conf:

[source::/var/log/foo]

Transforms must be applied in this
order to make sure events are dropped
on the floor prior to making their way
to the index processor

TRANSFORMS-set= setnull,setparsing
In transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = login
DEST_KEY = queue
FORMAT = indexQueue

[martin_mueller]

Do you have a question?