Splunk Search

Keep specific events and discard the rest

Log_wrangler
Builder

How to filter sets of monitored logs with HF?

Hi,

I have a number of logs files monitored by UFs and sent to autoLB HFs (which send downstream to the indexers).

I want to apply a regex filter to all the sources where I only collect logs that contain "error" or "warn".
Example search:

Index=common_index sourcetype=my_app_logs source = /temp/my_app/*

This will return a number of "source" field values where source =

/temp/my_app/logs/my_app.log
/temp/my_app/logs/my_app_gc.log
/temp/my_app/logs/my_app_user.log
/temp/my_app/logs/my_app_bootstart.log

On the UFs I have:

inputs.conf

[monitor:///temp/my_app/logs/*]
disabled = 0
index = common_index
sourcetype = my_app_logs

outputs.conf

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout-server://x.x.x.1:9997]

[tcpout:default-autolb-group]
disabled = false
server = x.x.x.1:9997,y.y.y.2:9997

[tcpout-server://y.y.y.2:9997]

On the HFs x.x.x.1, and y.y.y.2 I have:

outputs.conf
[tcpout]
defaultGroup = default-autolb-group
forwardedindex.filter.disable = true
indexAndForward = 0
maxQueueSize = 7MB

[tcpout-server://i.i.i.1:9997]

[tcpout-server://i.i.i.2:9997]

[tcpout:default-autolb-group]
disabled = false
server = i.i.i.1:9997,i.i.i.2:9997

Referencing [http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_even...]

It says to create "queue" stanzas in props and transforms (I am inferring on the HFs) this is where I need guidance.

For example, would I just add the following stanzas to the HFs?

in props.conf

[source::/temp/my_app/logs/*] <------- this is for applying the REGEX filter to all monitored logs
TRANSFORMS-set=setnull_myApp, setparsing_myApp

in transforms.conf

[setnull_myApp]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing_myApp]
REGEX = error, warn <------ ERROR OR WARN??
DEST_KEY = queue
FORMAT = indexQueue

My other question, if I want to selectively apply filters to some files (not all) source files would I create individual source stanzas?
For example:

[source::/temp/my_app/logs/my_app.log]

and would I need to change props and transforms individually as well?
For example:

[setnull_myApp.log]

[setparsing_myApp.log]

Thank you for your guidance!

0 Karma
1 Solution

somesoni2
Revered Legend

Your transforms.conf stanza should be this

[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue

You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.

View solution in original post

0 Karma

somesoni2
Revered Legend

Your transforms.conf stanza should be this

[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue

You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.

0 Karma

Log_wrangler
Builder

Thank you for your quick response, I will test your stanza and let you know.

0 Karma

Log_wrangler
Builder

It works! Thank you!!!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...