Getting Data In
Highlighted

Splunk not matching files with wildcard in monitor path in inputs.conf

New Member

I'm running splunk forwarder 6.4.1 on Ubuntu 14.04. I'm attempting to use splunk to monitor Jenkins build logs, which have a path like /var/lib/jenkins/jobs/*/builds/*/log, e.g. /var/lib/jenkins/jobs/ajobname/builds/1234/log, where log is a text file. My inputs.conf looks like this:

[monitor:///var/lib/jenkins/jobs/*/builds/*/log]
blacklist = \.(gz)$
recursive = False
sourcetype = jenkins
index = tools-jenkins
followSymlink = false

[monitor:///var/log]
blacklist = \.(gz)$
recursive = True
sourcetype = syslog
index = tools

Searching on the splunk server, I can see that logs from the second monitor stanza are getting indexed, while logs from the first stanza are not.

I checked the logs, and I see two relevant messages in splunkd.log:

06-27-2016 19:27:34.490 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/lib/jenkins/jobs/*/builds/*/log.
06-27-2016 19:27:34.491 +0000 INFO  TailingProcessor - Adding watch on path: /var/lib/jenkins/jobs.

I tried setting the monitor path on the first stanza to point to a specific file, which works as expected, which rules out potential permissions issues. I also tried using splunk list monitor to see the list of monitored files, but for some reason I'm unable to auth to use the CLI.

My best guess is that the first monitor stanza isn't matching the desired files. Why won't splunk monitor the jenkins logs?

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

Ultra Champion

What's the full name of the sample log file. Is it /var/lib/jenkins/jobs/ajobname/builds/1234/log?

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

New Member

The path of the file is /var/lib/jenkins/ajobname/builds/1234/ and the name of the file is log.

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

Ultra Champion

Interesting thing as -

asterisk

From Specify input paths with wildcards

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

New Member

Right, that's why I asked the question.

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

Ultra Champion

dunno - very weird...

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

Ultra Champion

Right ./splunk cmd btool inputs list monitor would probably show the /var/lib/jenkins/jobs/*/builds/*/log path as splunkd.log said - Parsing configuration stanza. But did it say Adding watch on path for that one?

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

Ultra Champion

Bug in Universal Forwarder? inputs.conf monitor and recursive = false

It speaks about a bug with the recursive option. Can you try without it?

I would actually strip it to a bare minimum, such as -

     [monitor:///var/lib/jenkins/jobs/*/builds/*/log]
     sourcetype = jenkins
     index = tools-jenkins
0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

New Member

Removing the recursive flag seems to have fixed the issue! If you make an answer, I will accept it.

0 Karma
Highlighted

Re: Splunk not matching files with wildcard in monitor path in inputs.conf

Ultra Champion

Wow - great to hear. Oh - let me make it a distinct answer.

0 Karma