Getting Data In

Unexpected precedence of monitor stanzas in inputs.conf

remeika
Explorer

I have two monitor stanzas to watch nginx access logs: a specific stanza to route a team's error logs to their specific index, and another fallback stanza to catch any error logs not routed to a specific index:

$ splunk cmd btool inputs list
...
[monitor:///var/log/nginx/*batman*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-batman
sourcetype = nginx-error
...
[monitor:///var/log/nginx/*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-fallback
sourcetype = nginx-error

My intention is that the file /var/log/nginx/batman-service-a-error.log is routed to index prod-batman, while the file /var/log/nginx/other-team-service-a-error.log is routed to prod-fallback. But this is not happening. I see:

$ splunk list monitor
Monitored Directories:
    ...
    /var/log/nginx/*error.log
        /var/log/nginx/batman-service-a-error.log
        /var/log/nginx/batman-service-b-error.log
        /var/log/nginx/batman-service-c-error.log

Indeed, there is no entry for /var/log/nginx/*batman*error.log in the output of splunk list monitor. Is there any way to force the stanza [monitor:///var/log/nginx/*batman*error.log] to take precedence over [monitor:///var/log/nginx/*error.log]?

0 Karma
1 Solution

micahkemp
Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

View solution in original post

christianvalin
Explorer

In network router configurations, the most specific match usually wins. As for inputs.conf "monitor" stanzas, its usually the least specific match that wins / take precendence if two or more match the wildcard. This makes huge difference in hostname matching when processing syslog directories, for example.

0 Karma

remeika
Explorer

for inputs.conf "monitor" stanzas, its usually the least specific match that
wins / take precendence if two or more match the wildcard

Could you be more specific about your use of the word "usually"? Is there any documentation to explain this?

0 Karma

micahkemp
Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

remeika
Explorer

Hopefully there is a cleaner way to do this out there, but this does work.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...