Splunk Search

Inputs.conf composed regex vs wildcard: Why is my whitelist monitor configuration not working?

splunk_zen
Builder

Why is this monitor whitelist not working ?

[monitor:///opt/logs/]
whitelist = (connectors/connectors\-\d\-boot|app1/app1\-\d\-boot|app2/app2(-stdout|-boot))\.log$

for these log files,

connectors/connectors-1-boot.log
app1/app1-1-boot.log
app2/app2-boot.log
app2/app2-stdout.log

whereas a wilcarded input is?

[monitor:///opt/logs/.../*-boot.log]
[monitor:///opt/logs/app2/app2-stdout.log]

If I split the problematic one into several ones it will pickup the corresponding files, eg

[monitor:///opt/logs/app2/]
whitelist = app2(-stdout|-boot)\.log$

Disclaimer, is it clashing with this other perfectly working monitor of mine ?
If it is, and the wildcard in the file input gets translated into an implicit whitelist anyway, why the difference ?

[monitor:///opt/logs/]
whitelist = (connectors/connectors\-.*|app2/app2(-server|-service))\.log$
blacklist = (boot|\.zip)
0 Karma
1 Solution

chanfoli
Builder

I think your identically named monitor stanzas could be creating confusion. Essentially you are defining the same monitor twice and splunk is probably going to use some strategy to determine a winning stanza.

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Give this a try

[monitor:///opt/logs/(connectors|app1|app2)/*]
whitelist = (connectors|app1|app2)/(connectors|app1|app2)(-\d+)*-(boot|stdout)\.log$
...Other settings...

OR

[monitor:///opt/logs/ (connectors|app1|app2)/(connectors|app1|app2)(-\d+)*-(boot|stdout)\.log]
...Other settings...
0 Karma

chanfoli
Builder

According to the current documentation at least, the above is not the way it is supposed to work.

From: http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Specifyinputpathswithwildcards

When determining the set of files or
directories to monitor, Splunk
Enterprise splits elements of a
monitoring stanza into segments -
defined as text between directory
separator characters ("/" or "\") in
the stanza definition. If you specify
a monitor stanza that contains
segments with both wildcards and regex
metacharacters (such as (, ), [, ],
and |), those characters behave
differently depending on where the
wild card is in the stanza.

...

If a monitoring stanza contains a
segment with regex metacharacters
before a segment with wildcards,
Splunk Enterprise treats the
metacharacters literally, as if you
wanted to monitor files or directories
with those characters in the files' or
directories' names.

...

If the regex metacharacters occur
within or after a segment that
contains a wildcard, Splunk Enterprise
treats the metacharacters as a regex
and matches files to monitor
accordingly.

splunk_zen
Builder

somesoni2, those didn't work.
I had previously tried similar wildcard + whitelist approaches and understood they won't play well together,
thus the fullly qualified path + whitelist approach (which was conflicting with another monitor over the same dir, but different file structures)

You are right Sean.

As soon as I tried an equivalent (but different) monitor stanza

[monitor:///opt/logs/.../]

Splunk accepted my composed regex whitelist.

chanfoli
Builder

I think your identically named monitor stanzas could be creating confusion. Essentially you are defining the same monitor twice and splunk is probably going to use some strategy to determine a winning stanza.

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...