Getting Data In

How do I edit my props and transforms to filter out logs that start with 30 and 32 in a local monitored directory on Windows?

grantsales
Engager

I'm using splunk enterprise on a local windows based system.

I have a file reader configured to watch a directory where I dump logs and folders of logs.

c:\logs\*\*.log

All folders and files that end in ".log"

There is a specific event that is typically in my .log files and they always start with 30 and 32. I'd like to filter this out and I've tried everything I can think of.

I even copied this type of setup, but I can't seem to get it working:
Section: "Discard specific events and keep the rest"
http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad

Used this for a reference for windows file paths:
http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Specifyinputpathswithwildcards

in the etc\system\local

props.conf

[source::....log]
TRANSFORMS-null= setnull

Also tried [source:://....log]
Also tried [monitor:://....log]
Also tried [monitor::....log]

transforms.conf

[setnull]
regex = ^3[02]
DEST_KEY = queue
FORMAT = nullQueue

After making changes, I restart splunk and send some test data, every time, my unwanted events that start with 30 and 32 still show up. Any help would be great, I'm pretty sure my regex is right, but I don't have any idea if the rest is.

Thanks,
Grant

0 Karma

yannK
Splunk Employee
Splunk Employee
  • First remark, do not use "setnull" as a transforms name, it's too generic ,and could overwrite an existing definition.
    Prefer something that describe better like : "setnull_logfilter"

  • Second remark, maybe a typo

    TRANSFROMS-null= setnull
    should be

    TRANSFORMS-null= setnull

grantsales
Engager

tried:

TRANSFORMS-null = setnull_dhcp

Also didn't work, I did change the transforms.conf file too when doing this name change.

0 Karma

yannK
Splunk Employee
Splunk Employee

The next step is to figure is you have a single instance or if this forwarder is sending data to another instance (indexer, or heavy forwarder)

The index time rules have to be setup on the instance that is parsing the events : the indexers (or the intermediary heavy forwarder if any)

example of forwarding architectures :
UF -> IDX (put rules here)
UF -> UF -> IDX (put rules here)
UF -> HF (put rules here) -> IDX
UF -> IDX (put rules here) -> IDX
IDX (put rules here)
HF (put rules here) -> IDX

0 Karma

grantsales
Engager

it's basically just a single instance test box, not forwarding any data.

0 Karma

yannK
Splunk Employee
Splunk Employee

Ok. so let's try with a broader props.conf condition

[source::*log]

0 Karma

grantsales
Engager

I found the input type by looking at etc\apps\search\local\inputs.conf

Turns out it's [monitor://C:\Logs\dhcplogs]

0 Karma

grantsales
Engager

Tried both with
TRANSFORMS-null = setnull_dhcp

Still not filtering correctly. Is my regex wrong? do I need to stick this into a different .conf file?

0 Karma

grantsales
Engager

How can I find out the correct type for the [source] or [monitor]?

0 Karma

grantsales
Engager

Yes, typo sorry.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...