Splunk Search

Keep specific events and discard the rest

Log_wrangler
Builder

How to filter sets of monitored logs with HF?

Hi,

I have a number of logs files monitored by UFs and sent to autoLB HFs (which send downstream to the indexers).

I want to apply a regex filter to all the sources where I only collect logs that contain "error" or "warn".
Example search:

Index=common_index sourcetype=my_app_logs source = /temp/my_app/*

This will return a number of "source" field values where source =

/temp/my_app/logs/my_app.log
/temp/my_app/logs/my_app_gc.log
/temp/my_app/logs/my_app_user.log
/temp/my_app/logs/my_app_bootstart.log

On the UFs I have:

inputs.conf

[monitor:///temp/my_app/logs/*]
disabled = 0
index = common_index
sourcetype = my_app_logs

outputs.conf

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout-server://x.x.x.1:9997]

[tcpout:default-autolb-group]
disabled = false
server = x.x.x.1:9997,y.y.y.2:9997

[tcpout-server://y.y.y.2:9997]

On the HFs x.x.x.1, and y.y.y.2 I have:

outputs.conf
[tcpout]
defaultGroup = default-autolb-group
forwardedindex.filter.disable = true
indexAndForward = 0
maxQueueSize = 7MB

[tcpout-server://i.i.i.1:9997]

[tcpout-server://i.i.i.2:9997]

[tcpout:default-autolb-group]
disabled = false
server = i.i.i.1:9997,i.i.i.2:9997

Referencing [http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_even...]

It says to create "queue" stanzas in props and transforms (I am inferring on the HFs) this is where I need guidance.

For example, would I just add the following stanzas to the HFs?

in props.conf

[source::/temp/my_app/logs/*] <------- this is for applying the REGEX filter to all monitored logs
TRANSFORMS-set=setnull_myApp, setparsing_myApp

in transforms.conf

[setnull_myApp]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing_myApp]
REGEX = error, warn <------ ERROR OR WARN??
DEST_KEY = queue
FORMAT = indexQueue

My other question, if I want to selectively apply filters to some files (not all) source files would I create individual source stanzas?
For example:

[source::/temp/my_app/logs/my_app.log]

and would I need to change props and transforms individually as well?
For example:

[setnull_myApp.log]

[setparsing_myApp.log]

Thank you for your guidance!

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Your transforms.conf stanza should be this

[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue

You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Your transforms.conf stanza should be this

[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue

You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.

0 Karma

Log_wrangler
Builder

Thank you for your quick response, I will test your stanza and let you know.

0 Karma

Log_wrangler
Builder

It works! Thank you!!!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...