Getting Data In

Unexpected precedence of monitor stanzas in inputs.conf

remeika
Explorer

I have two monitor stanzas to watch nginx access logs: a specific stanza to route a team's error logs to their specific index, and another fallback stanza to catch any error logs not routed to a specific index:

$ splunk cmd btool inputs list
...
[monitor:///var/log/nginx/*batman*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-batman
sourcetype = nginx-error
...
[monitor:///var/log/nginx/*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-fallback
sourcetype = nginx-error

My intention is that the file /var/log/nginx/batman-service-a-error.log is routed to index prod-batman, while the file /var/log/nginx/other-team-service-a-error.log is routed to prod-fallback. But this is not happening. I see:

$ splunk list monitor
Monitored Directories:
    ...
    /var/log/nginx/*error.log
        /var/log/nginx/batman-service-a-error.log
        /var/log/nginx/batman-service-b-error.log
        /var/log/nginx/batman-service-c-error.log

Indeed, there is no entry for /var/log/nginx/*batman*error.log in the output of splunk list monitor. Is there any way to force the stanza [monitor:///var/log/nginx/*batman*error.log] to take precedence over [monitor:///var/log/nginx/*error.log]?

0 Karma
1 Solution

micahkemp
Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

View solution in original post

christianvalin
Explorer

In network router configurations, the most specific match usually wins. As for inputs.conf "monitor" stanzas, its usually the least specific match that wins / take precendence if two or more match the wildcard. This makes huge difference in hostname matching when processing syslog directories, for example.

0 Karma

remeika
Explorer

for inputs.conf "monitor" stanzas, its usually the least specific match that
wins / take precendence if two or more match the wildcard

Could you be more specific about your use of the word "usually"? Is there any documentation to explain this?

0 Karma

micahkemp
Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

remeika
Explorer

Hopefully there is a cleaner way to do this out there, but this does work.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...