Hi
I'm attempting to configure my universal forwarder to read log files from a single directory with multiple subdirectories. We use log rotate so the files will be renamed with (1) up to (4) before starting again. I'm also trying to push those into the right index based on the file name. For example the top level directory is /srv/logs which has multiple subdirectories i.e
application
fileservice
proxyserver
each of these subdirectories contains multiple files from each environment (dev, int, prod etc) Here is an example file name.
application-prod.prod.log, i'm using the following inputs.conf which seems to work(ish). I've changed the monitor names to ensure they are treated as separate and i'm trying to blacklist anything I don't want to appear in each index.
[monitor:///srv/./logs]
blacklist = ppd..log$|prod..log$
sourcetype = service_log
index = nonprod
crcSalt =
[monitor:///srv/logs]
blacklist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = prod
crcSalt =
So in prod, I only want files that contain .prod and ppd, in nonprod I want devint, int, ft, infradev and nonprod.
So i'm wondering
- Are there better or more performant ways to configure these inputs
- Is there anyway I can check the data is correct in my indexes is correct (no prod data in non prod etc)
- If there are subdirectories should I be using recursive = true?
- The documentation says don't use crcSalt = with log rotate - however I see a number of initcrc errors - should I be setting a initcrclen = 2000 etc?
Sorry this is a long one, thanks for any help.
Thanks
Hi Stokers_23,
for my mental order, i prefer to have different stanzas for each kind of files, so I'd use this stanzas:
[monitor:///srv/logs/*/*.devint.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.int.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.infradev.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.nonprod.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.prod.log]
sourcetype = service_log
index = prod
disabled = 0
[monitor:///srv/logs/*/*.ppd.log]
sourcetype = service_log
index = prod
disabled = 0
Otherwise, if you want to reduce the number of stanzas you could use whitelist instead black list
[monitor:///srv/./logs]
whitelist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = nonprod
[monitor:///srv/logs]
whitelist = ppd..log$|prod..log$
sourcetype = service_log
index = prod
Answering to your questions:
index=prod | stats count by source
and verify the sources in your index, then repeat the same search for the nonprod indexI hope to be helpful for you.
Bye.
Giuseppe
Hi Stokers_23,
for my mental order, i prefer to have different stanzas for each kind of files, so I'd use this stanzas:
[monitor:///srv/logs/*/*.devint.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.int.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.infradev.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.nonprod.log]
sourcetype = service_log
index = nonprod
disabled = 0
[monitor:///srv/logs/*/*.prod.log]
sourcetype = service_log
index = prod
disabled = 0
[monitor:///srv/logs/*/*.ppd.log]
sourcetype = service_log
index = prod
disabled = 0
Otherwise, if you want to reduce the number of stanzas you could use whitelist instead black list
[monitor:///srv/./logs]
whitelist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = nonprod
[monitor:///srv/logs]
whitelist = ppd..log$|prod..log$
sourcetype = service_log
index = prod
Answering to your questions:
index=prod | stats count by source
and verify the sources in your index, then repeat the same search for the nonprod indexI hope to be helpful for you.
Bye.
Giuseppe
Hey, thanks for the detailed answer..
I have one issue left you might be able to help with, using the whitelists works really well. There is only one issue - the files named 'nonprod' are sometimes appearing in the prod index. I've tried blacklisting nonprod in the bottom stanza but that doesnt seem to work.
The file names contain '-prod.prod' so I could try the below?
[monitor:///srv/./logs]
whitelist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = nonprod
[monitor:///srv/logs]
whitelist = ppd..log$|-prod..log$
sourcetype = service_log
index = prod
Hi Stokers_23,
yes the problem is that prod
in contained in nonprod
, so you have to be mo detailed in your regex.
Bye.
Giuseppe