topic Re: How to whitelist files in directory and not in subdirectories? in Getting Data In

How to whitelist files in directory and not in subdirectories?

akchauhan — Tue, 29 Sep 2020 17:53:01 GMT

Hi Guys
I am trying to pick logs having job-info.*log name in common directory and job-heartbeat.*logs from heartbeat sub directory and job-error .*log from error sub directory. I used the configuration below and it is working fine. The only issue that I am facing is job-info.*log files are also getting picked up when someone places these files in other sub directories (xyz in below case). I don't want these files to pick when they are placed in any sub-directories, These should be picked only when they are placed in the common directory. Please suggest changes in the whitelist.

[monitor:///abc/common/]
disabled = false
index = infra_job
whitelist = (job-info.*log|heartbeat/job-heartbeat.*log|error/job-error.*log)

directories structure

cd /abc/common/

files/directories under this directory (example)

error/
heartbeat/
xyz/job-info.*log ---- don't want these logs to pick
job-info1.log
job-info2.log
job-info3.log

Thanks in advance

Re: How to whitelist files in directory and not in subdirectories?

ddrillic — Wed, 31 Jan 2018 19:56:31 GMT

recursive = false

might do it.

Re: How to whitelist files in directory and not in subdirectories?

horsefez — Wed, 31 Jan 2018 20:01:22 GMT

Hey akchauhan,

there is a blacklist setting you could set for the job-info*.log files.

Just add the following line to your config:

blacklist=(\/abc\/common\/xyz\/job-info\d*\.log)

This will make sure you definitely will not index files under the specified path, cuz blacklist take precedence over whitelists.
Stated here: If a file matches the regexes in both the blacklist and whitelist settings, the file is NOT monitored. Blacklists take precedence over whitelists.

Re: How to whitelist files in directory and not in subdirectories?

akchauhan — Thu, 01 Feb 2018 08:08:33 GMT

There are chances that dev team can create directory similar to xyz with other name as well. so what change do I need to add in blacklist you mentioned?

Re: How to whitelist files in directory and not in subdirectories?

akchauhan — Thu, 01 Feb 2018 12:05:15 GMT

this is not working. Adding recursive = false will not pick logs in error and heartbeat directory.

Re: How to whitelist files in directory and not in subdirectories?

FrankVl — Thu, 01 Feb 2018 13:07:12 GMT

Why not create 3 separate monitor inputs, one for job-info, one for error and one for heartbeat?

[monitor:///abc/common/job-info.*log]
disabled = false
index = infra_job

[monitor:///abc/common/heartbeat/job-heartbeat.*log]
disabled = false
index = infra_job

[monitor:///abc/common/error/job-error.*log]
disabled = false
index = infra_job

Re: How to whitelist files in directory and not in subdirectories?

akchauhan — Thu, 01 Feb 2018 13:36:35 GMT

yes, this was already in mind. Just thinking instead of creating 3 separate model inputs, if it is possible to do using whitelist/blacklist in single input that would be great.