topic Re: whitelist and wildcard for rotated file without recursive option in inputs.conf in Splunk Search

whitelist and wildcard for rotated file without recursive option in inputs.conf

koudis — Wed, 18 Jun 2014 11:38:52 GMT

Hi,
I have following configuration in inputs.conf:

[monitor:///var/log/audit/audit.log*]
whitelist=(audit\.log$|audit\.log.1$)
index=int-os
sourcetype=audit
recursive=false
[monitor:///mnt/log/messages*]
whitelist=(messages$|messages\.1$)
index=int-os
sourcetype=messages
recursive=false

Files are like this:

[root@instance:/home/milan.koudelka] ls -l /var/log/audit/
total 26604
-rw-r-----. 1 root splunk 2035776 Jun 18 13:31 audit.log
-r--r-----. 1 root splunk 6291677 Jun 17 17:37 audit.log.1
-r--r-----. 1 root splunk 6291484 Jun 15 02:09 audit.log.2
-r--r-----. 1 root splunk 6291614 Jun 12 10:28 audit.log.3
-r--r-----. 1 root splunk 6291656 Jun  9 19:09 audit.log.4

[root@splunk-ds1:/home/milan.koudelka] ls -l /mnt/log/messages*
-rw-r-----. 1 root splunk  18817 Jun 18 13:32 /mnt/log/messages
-rw-r-----. 1 root splunk  24468 Jun 18 03:42 /mnt/log/messages.1
-rw-r-----. 1 root splunk  52044 Jun 17 03:56 /mnt/log/messages.2

It's weird that for messages log file, input is working correctly, when for audit.log input isn't working at all.
I've tried to use all configurations like these:

[monitor:///var/log/audit/*]
[monitor:///var/log/audit/*.log]
[monitor:///var/log/audit/]

None of these are working. I don't want to allow recursive.

Only one working is

[monitor:///var/log/audit/audit.log]

But this will not catch first rotated file audit.log.1

Any advice why it's working for one log file and it isn't working for another ?

Splunk version 6.0.3

Re: whitelist and wildcard for rotated file without recursive option in inputs.conf

MuS — Wed, 18 Jun 2014 12:18:07 GMT

Hi koudis,

your whitelist for audit.log should be like this:

 whitelist=(audit\.log$|audit\.log\.1$)

cheers, MuS

Re: whitelist and wildcard for rotated file without recursive option in inputs.conf

koudis — Wed, 18 Jun 2014 12:38:20 GMT

Thx, for first tip MuS, but unfortunately this isn't working.
I'm sometimes using it without backslash by mistake like one any charater.

Weird is that it's working also for configurations like this

[monitor:///mnt/log/postgresql.log*]
whitelist=(.log$|.log.1$)
index=os
sourcetype=postgres
recursive=false

Re: whitelist and wildcard for rotated file without recursive option in inputs.conf

MuS — Wed, 18 Jun 2014 13:10:04 GMT

When you specify wildcards in a file input path, Splunk creates an implicit whitelist for that stanza. The longest fully qualified path becomes the monitor stanza, and the wildcards are translated into regular expressions.
This means your whitelist is being clobberd by your use of * expressions in the stanza.

Re: whitelist and wildcard for rotated file without recursive option in inputs.conf

koudis — Tue, 22 Jul 2014 15:50:20 GMT

So the easiest way to ensure that only two exact files (audit.log and audit.log.1) will be monitored, without any recursion is like this ?

[monitor:///var/log/audit/audit.log]
index=int-os
sourcetype=audit
recursive=false

[monitor:///var/log/audit/audit.log.1]
source=/var/log/audit/audit.log
index=int-os
sourcetype=audit
recursive=false