Getting Data In

How to configure my Marimba data input on a heavy forwarder to filter out some of the monitored logs?

nce054
Path Finder

I am currently trying to use my Marimba data gathered from the Endpoint tuner in Splunk. On my Universal Forwarder, I am placing a monitor on the log file I wish to monitor, and it is being sent correctly to the Indexer. I wish to filter these events on my Heavy Forwarder so that not all of them are sent, since there are certain ones I wish to ignore. Where do I start to do this? I believe it involves props.conf and transforms.conf, but I have had little interaction with both of them. Thanks for any help.

Edit
Here is an example from the Search Head. This is what is currently being received.

[16/Jun/2015:08:54:41 -0500] - warning nce054 50012 Common Reboot Service is disabled.
host = C235189 
index = main 
source = C:\Windows\.marimba\MarimbaEndpointTuner\history-y2015-m06-d16.log 
sourcetype =  history-y2015-m06-d-4

I was hoping to maybe only receive alerts with the first word being 'warning', like it is for this one.

EDIT
I have placed this in my props.conf on my Heavy Forwarder to see if it would label the data with a sourcetype.

[source::C:\\Windows\.marimba\MarimbaEndpointTuner\history...]
sourcetype = marimba
SHOULD_LINEMERGE = true

This didn't seem to have an effect. I'm wondering if I did the source incorrectly? AKA it's not finding anything that matches [source::C:\\Windows\.marimba\MarimbaEndpointTuner\history...]

0 Karma
1 Solution

lguinn2
Legend

First, I am unclear about your topology. Is it UF -> HF -> Indexer or UF -> Indexer? That makes a big difference! If you do have a heavy forwarder, why? Also, I think you are mis-using the ... in the props.conf.

In either topology, set the sourcetype in inputs.conf on the UF.

sourcetype=marimba

props.conf and transforms.conf go on the machine that is doing the parsing, either the HF or the indexer.

props.conf:

[marimba]
TRANSFORMS-mfilter=filter_marimba,remove_marimba

transforms.conf

[filter_marimba]
SOURCE_KEY=_raw
REGEX=\]\s-\swarning
DEST_KEY=_MetaData:Index
FORMAT=main

[remove_marimba]
SOURCE_KEY=_raw
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

This first filters out the data that you want to keep (warnings) and sends it to the main index. Then it sends all the remaining events to the trash.

View solution in original post

lguinn2
Legend

First, I am unclear about your topology. Is it UF -> HF -> Indexer or UF -> Indexer? That makes a big difference! If you do have a heavy forwarder, why? Also, I think you are mis-using the ... in the props.conf.

In either topology, set the sourcetype in inputs.conf on the UF.

sourcetype=marimba

props.conf and transforms.conf go on the machine that is doing the parsing, either the HF or the indexer.

props.conf:

[marimba]
TRANSFORMS-mfilter=filter_marimba,remove_marimba

transforms.conf

[filter_marimba]
SOURCE_KEY=_raw
REGEX=\]\s-\swarning
DEST_KEY=_MetaData:Index
FORMAT=main

[remove_marimba]
SOURCE_KEY=_raw
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

This first filters out the data that you want to keep (warnings) and sends it to the main index. Then it sends all the remaining events to the trash.

nce054
Path Finder

My setup is UF -> HF -> Indexer. I have now changed the inputs.conf on my UF to read `[monitor://C:\Windows.marimba\MarimbaEndpointTuner\history-y*.log]

index = marimba

sourcetype = marimba

disabled = 0
`
while the props.conf and transforms.conf on my HF have been changed to what you have showed me. Short of simply watching the Search Head and monitoring the inputs, is there an easier way to see if this is working?

0 Karma

nce054
Path Finder

Now that I have it configured, I'm still getting all of the messages. On the Search Head, the data is correctly labeled with index and sourcetype being 'marimba'. Shouldn't they be filtered though? I would expect that only ones with '] - warning' would appear.

0 Karma

lguinn2
Legend

It should be filtered, yes. But now I am thinking that maybe the props.conf and the transforms.conf should go on the indexer instead.

Also, remember that these changes only affect new data. Data that has already been indexed will not be changed.

0 Karma

nce054
Path Finder

I think that we were correct in assuming they should be on the Heavy Forwarder, since I want the Heavy Forwarder to lessen the amount of data being sent to the Indexer (aka filtering). And I give the change ample to take effect, but still sometimes the changes don't seem to have an effect.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...