Getting Data In

Unexpected precedence of monitor stanzas in inputs.conf

Explorer

I have two monitor stanzas to watch nginx access logs: a specific stanza to route a team's error logs to their specific index, and another fallback stanza to catch any error logs not routed to a specific index:

$ splunk cmd btool inputs list
...
[monitor:///var/log/nginx/*batman*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-batman
sourcetype = nginx-error
...
[monitor:///var/log/nginx/*error.log]
_rcvbuf = 1572864
host = p2....00a
index = prod-fallback
sourcetype = nginx-error

My intention is that the file /var/log/nginx/batman-service-a-error.log is routed to index prod-batman, while the file /var/log/nginx/other-team-service-a-error.log is routed to prod-fallback. But this is not happening. I see:

$ splunk list monitor
Monitored Directories:
    ...
    /var/log/nginx/*error.log
        /var/log/nginx/batman-service-a-error.log
        /var/log/nginx/batman-service-b-error.log
        /var/log/nginx/batman-service-c-error.log

Indeed, there is no entry for /var/log/nginx/*batman*error.log in the output of splunk list monitor. Is there any way to force the stanza [monitor:///var/log/nginx/*batman*error.log] to take precedence over [monitor:///var/log/nginx/*error.log]?

0 Karma
1 Solution

Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

View solution in original post

Explorer

In network router configurations, the most specific match usually wins. As for inputs.conf "monitor" stanzas, its usually the least specific match that wins / take precendence if two or more match the wildcard. This makes huge difference in hostname matching when processing syslog directories, for example.

0 Karma

Explorer

for inputs.conf "monitor" stanzas, its usually the least specific match that
wins / take precendence if two or more match the wildcard

Could you be more specific about your use of the word "usually"? Is there any documentation to explain this?

0 Karma

Champion

Put *batman*error.log in the blacklist for your *error.log stanza. From inputs.conf spec:

blacklist = <regular expression>
* If set, files from this input are NOT monitored if their path matches the
  specified regex.

View solution in original post

Explorer

Hopefully there is a cleaner way to do this out there, but this does work.

0 Karma