How to configure Props.conf and Transforms.conf for a sourcetype on my heavy forwarder to only send messages containing 'warning' to my indexer?

[nce054]

I am trying to alter how much data I am getting from my universal forwarder. The configuration I have is UF -> HF -> Indexer -> SH . I am getting a lot of data sent from the UF to the HF, and want to restrict what the HF sends to the indexer. Therefore, I am trying to create a filter using the props.conf and transforms.conf on the HF. So far, I have received some suggestions from the community, but I am still having troubles. I want only messages that contain 'warning' to be sent from the HF to the Indexer (for this certain sourcetype, that is). The sourcetype is 'marimba'. I will show you what I currently have in my configuration files.
inputs.conf

[monitor://C:\Windows\.marimba\MarimbaEndpointTuner\history-y*.log]
disabled=0
sourcetype = marimba

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba,remove-marimba

transforms.conf

[filter-marimba]
SOURCE_KEY=_raw
REGEX=\]\s-\swarning
DEST_KEY=_MetaData:Index
FORMAT=main

[remove-marimba]
SOURCE_KEY=_raw
REGEX= .
DEST_KEY=queue
FORMAT=nullQueue

However, I am still not getting anything on my search head. I have confirmed that if I take away the stanzas from props.conf and transforms.conf, I receive the data, with the index 'marimba' and sourcetype 'marimba'. Any ideas?

[tony_luu]

did you get the filtering to work?
i have the same issue, i.e. even REGEX=. didn't filter anything out
thanks,

[woodcock · Accepted solution]

I just tested this configuration and it DOES work:

[filter-marimba]
REGEX=^\[[^\]]+\]\s+-\s+(?!warning).*
DEST_KEY = queue
FORMAT = nullQueue

[woodcock]

Please click "Accept" if this worked.

[woodcock]

Try this:

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba

transforms.conf

[filter-marimba]
REGEX=(?!\]\s-\swarning)
DEST_KEY = queue
FORMAT = nullQueue

In this implementation, everything that MATCHES the REGEX is thrown away so if there is any problem, it is in my inversion of your original REGEX.

[nce054]

So it seems that this is allowing everything through now. Should it be throwing everything away but the 'warning's?

[woodcock]

So the problem is in the REGEX string. Show me an example of the data to keep and point out the distinctive vs. the data to trash and I will help make a better REGEX.

[nce054]

Good data
[18/Jun/2015:15:20:03 -0500] - warning nce054 50012 Common Reboot Service is disabled.

Bad data
#run script: SwapPrinters redirection: true
#script launch as user process
#ExeScript, redirect is true

[woodcock]

OK, try this (tested)

REGEX=^(?!\[[^\]]+\]\s+-\s+warning.*)

[nce054]

It's strange to me that all the data is somehow passing through. It's almost like it's totally disregarding the regex; even when I just put REGEX = . , the result is the same.

[woodcock]

Try this, just as a test (since most of your junk starts with #😞

REGEX = ^#

[nce054]

Still not having the desired effect. Thanks for your effort. I've got to leave work now, but I'll be picking this issue up again tomorrow.

[nce054]

Giving this a go. Will get back to you. Thanks.