Getting Data In

How to configure Props.conf and Transforms.conf for a sourcetype on my heavy forwarder to only send messages containing 'warning' to my indexer?

nce054
Path Finder

I am trying to alter how much data I am getting from my universal forwarder. The configuration I have is UF -> HF -> Indexer -> SH . I am getting a lot of data sent from the UF to the HF, and want to restrict what the HF sends to the indexer. Therefore, I am trying to create a filter using the props.conf and transforms.conf on the HF. So far, I have received some suggestions from the community, but I am still having troubles. I want only messages that contain 'warning' to be sent from the HF to the Indexer (for this certain sourcetype, that is). The sourcetype is 'marimba'. I will show you what I currently have in my configuration files.
inputs.conf

[monitor://C:\Windows\.marimba\MarimbaEndpointTuner\history-y*.log]
disabled=0
sourcetype = marimba

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba,remove-marimba

transforms.conf

[filter-marimba]
SOURCE_KEY=_raw
REGEX=\]\s-\swarning
DEST_KEY=_MetaData:Index
FORMAT=main

[remove-marimba]
SOURCE_KEY=_raw
REGEX= .
DEST_KEY=queue
FORMAT=nullQueue

However, I am still not getting anything on my search head. I have confirmed that if I take away the stanzas from props.conf and transforms.conf, I receive the data, with the index 'marimba' and sourcetype 'marimba'. Any ideas?

0 Karma
1 Solution

woodcock
Esteemed Legend

I just tested this configuration and it DOES work:

[filter-marimba]
REGEX=^\[[^\]]+\]\s+-\s+(?!warning).*
DEST_KEY = queue
FORMAT = nullQueue

View solution in original post

tony_luu
Path Finder

did you get the filtering to work?
i have the same issue, i.e. even REGEX=. didn't filter anything out
thanks,

0 Karma

woodcock
Esteemed Legend

I just tested this configuration and it DOES work:

[filter-marimba]
REGEX=^\[[^\]]+\]\s+-\s+(?!warning).*
DEST_KEY = queue
FORMAT = nullQueue

woodcock
Esteemed Legend

Please click "Accept" if this worked.

0 Karma

woodcock
Esteemed Legend

Try this:

props.conf

[marimba]
TRANSFORMS-mfilter=filter-marimba

transforms.conf

[filter-marimba]
REGEX=(?!\]\s-\swarning)
DEST_KEY = queue
FORMAT = nullQueue

In this implementation, everything that MATCHES the REGEX is thrown away so if there is any problem, it is in my inversion of your original REGEX.

nce054
Path Finder

So it seems that this is allowing everything through now. Should it be throwing everything away but the 'warning's?

0 Karma

woodcock
Esteemed Legend

So the problem is in the REGEX string. Show me an example of the data to keep and point out the distinctive vs. the data to trash and I will help make a better REGEX.

0 Karma

nce054
Path Finder

Good data
[18/Jun/2015:15:20:03 -0500] - warning nce054 50012 Common Reboot Service is disabled.

Bad data
#run script: SwapPrinters redirection: true
#script launch as user process
#ExeScript, redirect is true

0 Karma

woodcock
Esteemed Legend

OK, try this (tested)

REGEX=^(?!\[[^\]]+\]\s+-\s+warning.*)
0 Karma

nce054
Path Finder

It's strange to me that all the data is somehow passing through. It's almost like it's totally disregarding the regex; even when I just put REGEX = . , the result is the same.

0 Karma

woodcock
Esteemed Legend

Try this, just as a test (since most of your junk starts with #😞

REGEX = ^#
0 Karma

nce054
Path Finder

Still not having the desired effect. Thanks for your effort. I've got to leave work now, but I'll be picking this issue up again tomorrow.

0 Karma

nce054
Path Finder

Giving this a go. Will get back to you. Thanks.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...