How to filter sets of monitored logs with HF?
Hi,
I have a number of logs files monitored by UFs and sent to autoLB HFs (which send downstream to the indexers).
I want to apply a regex filter to all the sources where I only collect logs that contain "error" or "warn".
Example search:
Index=common_index sourcetype=my_app_logs source = /temp/my_app/*
This will return a number of "source" field values where source =
/temp/my_app/logs/my_app.log
/temp/my_app/logs/my_app_gc.log
/temp/my_app/logs/my_app_user.log
/temp/my_app/logs/my_app_bootstart.log
On the UFs I have:
inputs.conf
[monitor:///temp/my_app/logs/*]
disabled = 0
index = common_index
sourcetype = my_app_logs
outputs.conf
[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0
[tcpout-server://x.x.x.1:9997]
[tcpout:default-autolb-group]
disabled = false
server = x.x.x.1:9997,y.y.y.2:9997
[tcpout-server://y.y.y.2:9997]
On the HFs x.x.x.1, and y.y.y.2 I have:
outputs.conf
[tcpout]
defaultGroup = default-autolb-group
forwardedindex.filter.disable = true
indexAndForward = 0
maxQueueSize = 7MB
[tcpout-server://i.i.i.1:9997]
[tcpout-server://i.i.i.2:9997]
[tcpout:default-autolb-group]
disabled = false
server = i.i.i.1:9997,i.i.i.2:9997
Referencing [http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Keep_specific_even...]
It says to create "queue" stanzas in props and transforms (I am inferring on the HFs) this is where I need guidance.
For example, would I just add the following stanzas to the HFs?
in props.conf
[source::/temp/my_app/logs/*] <------- this is for applying the REGEX filter to all monitored logs
TRANSFORMS-set=setnull_myApp, setparsing_myApp
in transforms.conf
[setnull_myApp]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing_myApp]
REGEX = error, warn <------ ERROR OR WARN??
DEST_KEY = queue
FORMAT = indexQueue
My other question, if I want to selectively apply filters to some files (not all) source files would I create individual source stanzas?
For example:
[source::/temp/my_app/logs/my_app.log]
and would I need to change props and transforms individually as well?
For example:
[setnull_myApp.log]
[setparsing_myApp.log]
Thank you for your guidance!
Your transforms.conf stanza should be this
[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue
You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.
Your transforms.conf stanza should be this
[setparsing_myApp]
REGEX = (?i)(error|warn)
DEST_KEY = queue
FORMAT = indexQueue
You can create a dedicated props.conf stanza for selected sources to ensure it's applied wherever you need. You can re-use your transforms.conf stanza as long as the source you're applying it to requires same regex/filtering rules.
Thank you for your quick response, I will test your stanza and let you know.
It works! Thank you!!!