How to use wildcards and whitelists in monitor stanza (inputs.conf)?

[plunkingalex]

I'm using a deployment server to distribute a single inputs.conf file to a number of servers in a class. The locations of the files that I need to monitor are similar between the servers, but sometimes (sub)directories refer to the servers instead of being generically named. This circumstance made me reach for wildcards / whitelists in determining the paths of the files to watch. (The alternative would be creating separate monitor stanzas for each individual server in the class, which defeats the purpose.) Can't get it to work, though. What am I missing?

These are the directories / files on the various servers I want to monitor:

/base/logs/appl/xxx.seg.ex/logfile1.log
/base/logs/appl/xxx.seg.ex/logfile2.log
/base/logs/appl/yyy.seg.ex/logfile1.log
/base/logs/appl/yyy.seg.ex/logfile2.log

And these are the monitor stanzas I'd set up in inputs.conf:

[monitor:///base/logs/appl/*.seg.ex/logfile1.log]
index=index

[monitor:///base/logs/appl/*.seg.ex/logfile2.log]
index=index

Unfortunately this does not work...

Checking the _internal index made clear that the monitor stanzas are not OK. Apparently implicit whitelists were added:

'^\/base\/logs\/appl/[^/]*.seg.ex/logfile1.log$' (on path 'monitor:///base/logs/appl') [1]

'^\/base\/logs\/appl/[^/]*.seg.ex/logfile2.log$' (on path 'monitor:///base/logs/appl') [2]

The _internal index also contains logevents saying:

TailingProcessor - Will not call watch on path '/base/logs/appl/xxx.seg.ex/logfile1.log due to stanza: monitor:///base/logs/appl/*.seg.ex/logfile1.log [1]

TailingProcessor - Will not call watch on path '/base/logs/appl/xxx.seg.ex/logfile2.log due to stanza: monitor:///base/logs/appl/*.seg.ex/logfile1.log [2]

Why doesn't this work? And how could I get it to work as desired?

[codebuilder]

The problem is that because of the wildcard, you are telling the forwarder to essentially monitor the same directory and files, but send them to two different indexes.

You'll need to further delimit the directory or file names being monitored so that they are unique.
Splunk can send the same file to two different indexes, but not using the configuration that you have in place.

[plunkingalex]

Will try to set-up an instance where I can test; want to see if I can replicate this behaviour and rule out a specific issue with this environment.

[codebuilder]

I'm not sure what your filesystem structure looks like, but it could be a recursive issue.

Try using this:
[monitor:///base/logs/appl/.../logfile1.log]

[plunkingalex]

Tried inputs.conf with only one monitor stanza, which I also simplified by replacing a whole segment of the path with an * (instead using the wildcard for a part of a segment):

[monitor:///base/logs/appl/*/logfile1.log]
index=index

This configuration is almost exactly the same as one shown in the examples here: https://docs.splunk.com/Documentation/Splunk/8.0.3/Data/Specifyinputpathswithwildcards

Strangely, it still doen't work. The same / similar errors in _internal.

[codebuilder]

Did you cycle Splunk after making the changes?

[plunkingalex]

If by 'cycle', you mean 'restart the forwarder' then yes. I performed a restart of the forwarder every time I changed the inputs.config file.

[plunkingalex]

Tried inputs.conf without the second monitor stanza, so there was only:

[monitor:///base/logs/appl/*.seg.ex/logfile1.log]
index=index

Didn't work; the same / similar events pop up in the _internal index.

[plunkingalex]

Thanks for your reply, codebuilder. Changed the index stanza so that both files go to the same index, but I still get the same errors. What gives?!