Bug in Universal Forwarder? inputs.conf monitor and recursive = false

[elof]

Should it really be like this? I think it is a bug.

In /var/log I have lots of files and dirs.
I want to monitor the files in /var/log, but I don't want to recurse through all the subdirs. I'm only interested in a few specific subfolders.

In inputs.conf of my universal forwarder (on a Debian machine) I have stanzas like this:

### Files in /var/log
[monitor:///var/log]
disabled = false
index = foo
sourcetype = syslog
ignoreOlderThan = 14d
recursive = false
blacklist = (\.gz|\.\d|/all.log|/faillog|/lastlog|/wtmp)$

### Apache logs
[monitor:///var/log/apache2]
disabled = false
index = foo
ignoreOlderThan = 14d
recursive = false
blacklist = (\.gz|\.\d)$

### qqq logs
[monitor:///var/log/qqq]
disabled = false
index = foo
sourcetype = qqq
ignoreOlderThan = 14d
recursive = false
blacklist = (\.gz|\.\d)$

Bug/problem:
Only the files in /var/log (such as /var/log/messages) get indexed! The UF is not starting to monitor the apache2 nor qqq directories. 😞

Apparently the option "recursive = false" in the first stanza for /var/log is not constrained to that stanza, but also put restrictions on all other stanzas related to the /var/log path.

If I comment out the first stanza (or its option "recursive = false"), the other stanzas start working.

Have I missed some vital keyword/option in the manual, 'cause this can't be how Splunk is supposed to behave, is it?

(both Debian and the UF use the latest version)

[jayavardhand]

This question of yours helped me identify an issue which I had no clue of why when I set recursive = false on a sub folder doesn't monitor files in that folder. I've noticed a parent folder setting of recursive = false and that appears to be like an obvious issue.

[neelamssantosh]

why to give 'blacklist' of Specific extensions of compressed files to exclude, where splunk ignores
packed_extensions_list:
bz, bz2, tbz, tbz2, Z, gz, tgz, tar, zip

[elof · Accepted solution]

Not a solution to this bug, but a workaround:

I removed all occurrances of the problematic option recursive.
In its place I did two changes:
* instead of monitoring a dir, /var/log, I monitor the items within the dir, /var/log/*
* I added a blacklist regexp filter to exclude any and all subfolders, /var/log/.*/

Result: I monitor all files in the dir but don't recurse into any subfolders. In my two subfolders of interest I do the same.

Example:

[monitor:///var/log/*]
blacklist = /var/log/.*/
blacklist = (\.gz|\.\d|/all.log|/faillog|/lastlog|/wtmp)$
disabled = false
index = foo
sourcetype = syslog
ignoreOlderThan = 14d

### Apache logs
[monitor:///var/log/apache2/*]
blacklist = /var/log/apache2/.*/
blacklist = (\.gz|\.\d)$
disabled = false
index = foo
ignoreOlderThan = 14d

### qqq logs
[monitor:///var/log/qqq/*]
blacklist = /var/log/qqq/.*/
blacklist = (\.gz|\.\d)$
disabled = false
index = foo
sourcetype = qqq
ignoreOlderThan = 14d