Getting Data In

How to configure Props.conf and Transforms.conf for a sourcetype on my heavy forwarder to only send messages containing 'warning' to my indexer?

Path Finder

I am trying to alter how much data I am getting from my universal forwarder. The configuration I have is UF -> HF -> Indexer -> SH . I am getting a lot of data sent from the UF to the HF, and want to restrict what the HF sends to the indexer. Therefore, I am trying to create a filter using the props.conf and transforms.conf on the HF. So far, I have received some suggestions from the community, but I am still having troubles. I want only messages that contain 'warning' to be sent from the HF to the Indexer (for this certain sourcetype, that is). The sourcetype is 'marimba'. I will show you what I currently have in my configuration files.
inputs.conf

[monitor://C:\Windows\.marimba\MarimbaEndpointTuner\history-y*.log]
disabled=0
sourcetype = marimba

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba,remove-marimba

transforms.conf

[filter-marimba]
SOURCE_KEY=_raw
REGEX=\]\s-\swarning
DEST_KEY=_MetaData:Index
FORMAT=main

[remove-marimba]
SOURCE_KEY=_raw
REGEX= .
DEST_KEY=queue
FORMAT=nullQueue

However, I am still not getting anything on my search head. I have confirmed that if I take away the stanzas from props.conf and transforms.conf, I receive the data, with the index 'marimba' and sourcetype 'marimba'. Any ideas?

0 Karma
1 Solution

Esteemed Legend

I just tested this configuration and it DOES work:

[filter-marimba]
REGEX=^\[[^\]]+\]\s+-\s+(?!warning).*
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

Path Finder

did you get the filtering to work?
i have the same issue, i.e. even REGEX=. didn't filter anything out
thanks,

0 Karma

Esteemed Legend

I just tested this configuration and it DOES work:

[filter-marimba]
REGEX=^\[[^\]]+\]\s+-\s+(?!warning).*
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

Esteemed Legend

Please click "Accept" if this worked.

0 Karma

Esteemed Legend

Try this:

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba

transforms.conf

[filter-marimba]
REGEX=(?!\]\s-\swarning)
DEST_KEY = queue
FORMAT = nullQueue

In this implementation, everything that MATCHES the REGEX is thrown away so if there is any problem, it is in my inversion of your original REGEX.

Path Finder

So it seems that this is allowing everything through now. Should it be throwing everything away but the 'warning's?

0 Karma

Esteemed Legend

So the problem is in the REGEX string. Show me an example of the data to keep and point out the distinctive vs. the data to trash and I will help make a better REGEX.

0 Karma

Path Finder

Good data
[18/Jun/2015:15:20:03 -0500] - warning nce054 50012 Common Reboot Service is disabled.

Bad data
#run script: SwapPrinters redirection: true
#script launch as user process
#ExeScript, redirect is true

0 Karma

Esteemed Legend

OK, try this (tested)

REGEX=^(?!\[[^\]]+\]\s+-\s+warning.*)
0 Karma

Path Finder

It's strange to me that all the data is somehow passing through. It's almost like it's totally disregarding the regex; even when I just put REGEX = . , the result is the same.

0 Karma

Esteemed Legend

Try this, just as a test (since most of your junk starts with #😞

REGEX = ^#
0 Karma

Path Finder

Still not having the desired effect. Thanks for your effort. I've got to leave work now, but I'll be picking this issue up again tomorrow.

0 Karma

Path Finder

Giving this a go. Will get back to you. Thanks.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!