Splunk not matching files with wildcard in monitor path in inputs.conf

[max_edx]

I'm running splunk forwarder 6.4.1 on Ubuntu 14.04. I'm attempting to use splunk to monitor Jenkins build logs, which have a path like /var/lib/jenkins/jobs/*/builds/*/log, e.g. /var/lib/jenkins/jobs/ajobname/builds/1234/log, where log is a text file. My inputs.conf looks like this:

[monitor:///var/lib/jenkins/jobs/*/builds/*/log]
blacklist = \.(gz)$
recursive = False
sourcetype = jenkins
index = tools-jenkins
followSymlink = false

[monitor:///var/log]
blacklist = \.(gz)$
recursive = True
sourcetype = syslog
index = tools

Searching on the splunk server, I can see that logs from the second monitor stanza are getting indexed, while logs from the first stanza are not.

I checked the logs, and I see two relevant messages in splunkd.log:

06-27-2016 19:27:34.490 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/lib/jenkins/jobs/*/builds/*/log.
06-27-2016 19:27:34.491 +0000 INFO  TailingProcessor - Adding watch on path: /var/lib/jenkins/jobs.

I tried setting the monitor path on the first stanza to point to a specific file, which works as expected, which rules out potential permissions issues. I also tried using splunk list monitor to see the list of monitored files, but for some reason I'm unable to auth to use the CLI.

My best guess is that the first monitor stanza isn't matching the desired files. Why won't splunk monitor the jenkins logs?

[ddrillic · Accepted solution]

Let's please remove the recursive = false option. Apparently it's buggy ...

[ddrillic]

max_edx, please accept the answer if indeed it solved the issue.

[ddrillic]

What's the full name of the sample log file. Is it /var/lib/jenkins/jobs/ajobname/builds/1234/log?

[max_edx]

The path of the file is /var/lib/jenkins/ajobname/builds/1234/ and the name of the file is log.

[ddrillic]

Interesting thing as -

From Specify input paths with wildcards

[max_edx]

Right, that's why I asked the question.

[ddrillic]

dunno - very weird...

[ddrillic]

Right ./splunk cmd btool inputs list monitor would probably show the /var/lib/jenkins/jobs/*/builds/*/log path as splunkd.log said - Parsing configuration stanza. But did it say Adding watch on path for that one?

[ddrillic]

Bug in Universal Forwarder? inputs.conf monitor and recursive = false

It speaks about a bug with the recursive option. Can you try without it?

I would actually strip it to a bare minimum, such as -

     [monitor:///var/lib/jenkins/jobs/*/builds/*/log]
     sourcetype = jenkins
     index = tools-jenkins

[max_edx]

Removing the recursive flag seems to have fixed the issue! If you make an answer, I will accept it.

[ddrillic]

Wow - great to hear. Oh - let me make it a distinct answer.