Getting Data In

Splunk not matching files with wildcard in monitor path in inputs.conf

max_edx
New Member

I'm running splunk forwarder 6.4.1 on Ubuntu 14.04. I'm attempting to use splunk to monitor Jenkins build logs, which have a path like /var/lib/jenkins/jobs/*/builds/*/log, e.g. /var/lib/jenkins/jobs/ajobname/builds/1234/log, where log is a text file. My inputs.conf looks like this:

[monitor:///var/lib/jenkins/jobs/*/builds/*/log]
blacklist = \.(gz)$
recursive = False
sourcetype = jenkins
index = tools-jenkins
followSymlink = false

[monitor:///var/log]
blacklist = \.(gz)$
recursive = True
sourcetype = syslog
index = tools

Searching on the splunk server, I can see that logs from the second monitor stanza are getting indexed, while logs from the first stanza are not.

I checked the logs, and I see two relevant messages in splunkd.log:

06-27-2016 19:27:34.490 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/lib/jenkins/jobs/*/builds/*/log.
06-27-2016 19:27:34.491 +0000 INFO  TailingProcessor - Adding watch on path: /var/lib/jenkins/jobs.

I tried setting the monitor path on the first stanza to point to a specific file, which works as expected, which rules out potential permissions issues. I also tried using splunk list monitor to see the list of monitored files, but for some reason I'm unable to auth to use the CLI.

My best guess is that the first monitor stanza isn't matching the desired files. Why won't splunk monitor the jenkins logs?

0 Karma
1 Solution

ddrillic
Ultra Champion

Let's please remove the recursive = false option. Apparently it's buggy ...

View solution in original post

ddrillic
Ultra Champion

Let's please remove the recursive = false option. Apparently it's buggy ...

View solution in original post

ddrillic
Ultra Champion

max_edx, please accept the answer if indeed it solved the issue.

0 Karma

ddrillic
Ultra Champion

What's the full name of the sample log file. Is it /var/lib/jenkins/jobs/ajobname/builds/1234/log?

0 Karma

max_edx
New Member

The path of the file is /var/lib/jenkins/ajobname/builds/1234/ and the name of the file is log.

0 Karma

ddrillic
Ultra Champion

Interesting thing as -

asterisk

From Specify input paths with wildcards

0 Karma

max_edx
New Member

Right, that's why I asked the question.

0 Karma

ddrillic
Ultra Champion

dunno - very weird...

0 Karma

ddrillic
Ultra Champion

Right ./splunk cmd btool inputs list monitor would probably show the /var/lib/jenkins/jobs/*/builds/*/log path as splunkd.log said - Parsing configuration stanza. But did it say Adding watch on path for that one?

0 Karma

ddrillic
Ultra Champion

Bug in Universal Forwarder? inputs.conf monitor and recursive = false

It speaks about a bug with the recursive option. Can you try without it?

I would actually strip it to a bare minimum, such as -

     [monitor:///var/lib/jenkins/jobs/*/builds/*/log]
     sourcetype = jenkins
     index = tools-jenkins
0 Karma

max_edx
New Member

Removing the recursive flag seems to have fixed the issue! If you make an answer, I will accept it.

0 Karma

ddrillic
Ultra Champion

Wow - great to hear. Oh - let me make it a distinct answer.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!