Getting Data In

Top level monitor wildcard Linux

Splunk Employee
Splunk Employee

I am working in an environment with an unfortunate naming standard. Essentially on ever app server the path to common logs is obfuscated by variability in multiple places in the path, most problematically in the top level director below root. This is causing my monitor stanza to not capture events as desired.

Sample paths:
/jdufowral1/log/apps/INSTANCE1/application_CM1/tomcat.out
/jdufowral1/log/apps/INSTANCE1/application_CM2/tomcat.out
/hdufowdal3/log/apps/INSTANCE1/application_CM1/tomcat.out

etc etc etc.

the monitor I was attempting was based on the following.
[monitor:///.../application_CM*/]

with whitelists to limit the captured files. It has been discussed also using whitelists to regex limit the top level directories that are search (essence of \w{9}\d or some such). The problem is that the monitor is not capturing any files.

A similar monitor written as
[monitor:///jdufowral1/.../application_CM*/]

works fine. It seems that a wildcard in a top level directory isn't working. I've seen mention of this related to windows splunk instances, but as mentioned this is a linux environment.

My request is basically either a way to wildcard from top level directories, or an explanation as to why I can't. I can work out a workaround I'm sure, but since the goal was to limit the number of inputs.conf stanzas I'd like to at least understand why I have to make something less generic.

I've also tried tricky monitor attempts to attach a top level directory, also no good results.
[monitor:///opt/./.../application_CM*]

1 Solution

Splunk Employee
Splunk Employee

The general consensus seems to be that the language that evaluates the monitor stanza requires a literal path point to begin at. I had assumed that monitor:/// was explicitly setting / as the begin point, but in actuality the final / is interpreted as an implied starting point, as pwd in / instead of / explicitly for instance.

The solution we found for situations where the log path needed to begin with a wildcard was to create a symbolic link in a static position that linked to /.
e.g. #ln-s / /var/splunk
Then [monitor:///var/splunk/.../application_CM*]

The only consideration to this is the complication caused by sym links and crcSalt.
see http://splunk-base.splunk.com/answers/38278/duplicate-log-entries-with-crcsalt-and-symbolic-links

The short of it is, if you use the sym link as the only path by which those logs will be evaluated, crcSalt will be fine. If you have multiple sym links or a monitor that can reach the direct file, the crcSalt will see them as different files.

View solution in original post

New Member

Question about this solution.

Doesn't the /.../ start at the current directory? So in the example above:

monitor:///var/splunk/.../application_CM*]

will not move "up" to the same level as the new symlink: /var/splunk, instead will look within (and down) the /var/splunk/ directory.
Is there a way to move UP within the monitor stanza?

0 Karma

Splunk Employee
Splunk Employee

I believe the language that the monitor stanza uses would read a single . as parent directory. IE

[monitor:///var/splunk/..././]

0 Karma

Splunk Employee
Splunk Employee

The general consensus seems to be that the language that evaluates the monitor stanza requires a literal path point to begin at. I had assumed that monitor:/// was explicitly setting / as the begin point, but in actuality the final / is interpreted as an implied starting point, as pwd in / instead of / explicitly for instance.

The solution we found for situations where the log path needed to begin with a wildcard was to create a symbolic link in a static position that linked to /.
e.g. #ln-s / /var/splunk
Then [monitor:///var/splunk/.../application_CM*]

The only consideration to this is the complication caused by sym links and crcSalt.
see http://splunk-base.splunk.com/answers/38278/duplicate-log-entries-with-crcsalt-and-symbolic-links

The short of it is, if you use the sym link as the only path by which those logs will be evaluated, crcSalt will be fine. If you have multiple sym links or a monitor that can reach the direct file, the crcSalt will see them as different files.

View solution in original post

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!