topic Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows? in Getting Data In

How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 16:35:40 GMT

I'm using splunk enterprise on a local windows based system.

I have a file reader configured to watch a directory where I dump logs and folders of logs.

c:\logs\*\*.log

All folders and files that end in ".log"

There is a specific event that is typically in my .log files and they always start with 30 and 32. I'd like to filter this out and I've tried everything I can think of.

I even copied this type of setup, but I can't seem to get it working:
Section: "Discard specific events and keep the rest"
http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad

Used this for a reference for windows file paths:
http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Specifyinputpathswithwildcards

in the etc\system\local

props.conf

[source::....log]
TRANSFORMS-null= setnull

Also tried [source:://....log]
Also tried [monitor:://....log]
Also tried [monitor::....log]

transforms.conf

[setnull]
regex = ^3[02]
DEST_KEY = queue
FORMAT = nullQueue

After making changes, I restart splunk and send some test data, every time, my unwanted events that start with 30 and 32 still show up. Any help would be great, I'm pretty sure my regex is right, but I don't have any idea if the rest is.

Thanks,
Grant

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

yannK — Mon, 27 Apr 2015 16:42:45 GMT

First remark, do not use "setnull" as a transforms name, it's too generic ,and could overwrite an existing definition.
Prefer something that describe better like : "setnull_logfilter"
Second remark, maybe a typo

TRANSFROMS-null= setnull
should be

TRANSFORMS-null= setnull

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 16:44:56 GMT

Yes, typo sorry.

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 16:46:28 GMT

How can I find out the correct type for the [source] or [monitor]?

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 16:51:23 GMT

tried:

TRANSFORMS-null = setnull_dhcp

Also didn't work, I did change the transforms.conf file too when doing this name change.

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

yannK — Mon, 27 Apr 2015 17:10:12 GMT

The next step is to figure is you have a single instance or if this forwarder is sending data to another instance (indexer, or heavy forwarder)

The index time rules have to be setup on the instance that is parsing the events : the indexers (or the intermediary heavy forwarder if any)

example of forwarding architectures :
UF -> IDX (put rules here)
UF -> UF -> IDX (put rules here)
UF -> HF (put rules here) -> IDX
UF -> IDX (put rules here) -> IDX
IDX (put rules here)
HF (put rules here) -> IDX

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 17:18:05 GMT

it's basically just a single instance test box, not forwarding any data.

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

yannK — Mon, 27 Apr 2015 17:22:27 GMT

Ok. so let's try with a broader props.conf condition

[source::*log]

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 17:29:30 GMT

I found the input type by looking at etc\apps\search\local\inputs.conf

Turns out it's [monitor://C:\Logs\dhcplogs]

Re: How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales — Mon, 27 Apr 2015 17:31:42 GMT

Tried both with
TRANSFORMS-null = setnull_dhcp

Still not filtering correctly. Is my regex wrong? do I need to stick this into a different .conf file?