Keep specific events and discard the rest

Log_wrangler — Tue, 29 Sep 2020 19:22:00 GMT

How to filter sets of monitored logs with HF?

Hi,

I have a number of logs files monitored by UFs and sent to autoLB HFs (which send downstream to the indexers).

I want to apply a regex filter to all the sources where I only collect logs that contain "error" or "warn".
Example search:

Index=common_index sourcetype=my_app_logs source = /temp/my_app/*

This will return a number of "source" field values where source =

/temp/my_app/logs/my_app.log
/temp/my_app/logs/my_app_gc.log
/temp/my_app/logs/my_app_user.log
/temp/my_app/logs/my_app_bootstart.log

On the UFs I have:

inputs.conf

[monitor:///temp/my_app/logs/*]
disabled = 0
index = common_index
sourcetype = my_app_logs

outputs.conf

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout-server://x.x.x.1:9997]

[tcpout:default-autolb-group]
disabled = false
server = x.x.x.1:9997,y.y.y.2:9997

[tcpout-server://y.y.y.2:9997]

On the HFs x.x.x.1, and y.y.y.2 I have:

outputs.conf
[tcpout]
defaultGroup = default-autolb-group
forwardedindex.filter.disable = true
indexAndForward = 0
maxQueueSize = 7MB

[tcpout-server://i.i.i.1:9997]

[tcpout-server://i.i.i.2:9997]

[tcpout:default-autolb-group]
disabled = false
server = i.i.i.1:9997,i.i.i.2:9997

Referencing [http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest]

It says to create "queue" stanzas in props and transforms (I am inferring on the HFs) this is where I need guidance.

For example, would I just add the following stanzas to the HFs?

in props.conf

[source::/temp/my_app/logs/*] <------- this is for applying the REGEX filter to all monitored logs
TRANSFORMS-set=setnull_myApp, setparsing_myApp

in transforms.conf

[setnull_myApp]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing_myApp]
REGEX = error, warn <------ ERROR OR WARN??
DEST_KEY = queue
FORMAT = indexQueue

My other question, if I want to selectively apply filters to some files (not all) source files would I create individual source stanzas?
For example:

[source::/temp/my_app/logs/my_app.log]

and would I need to change props and transforms individually as well?
For example:

[setnull_myApp.log]

[setparsing_myApp.log]

Thank you for your guidance!

Re: Keep specific events and discard the rest

somesoni2 — Mon, 07 May 2018 15:05:46 GMT

Your transforms.conf stanza should be this

[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue

You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.

Re: Keep specific events and discard the rest

Log_wrangler — Mon, 07 May 2018 15:35:11 GMT

Thank you for your quick response, I will test your stanza and let you know.

Re: Keep specific events and discard the rest

Log_wrangler — Tue, 08 May 2018 21:36:09 GMT

It works! Thank you!!!

topic Re: Keep specific events and discard the rest in Splunk Search

Keep specific events and discard the rest

Re: Keep specific events and discard the rest

Re: Keep specific events and discard the rest

Re: Keep specific events and discard the rest