topic Re: Universal forwarder - multiple inputs.conf stanzas on the same folder in Getting Data In

Universal forwarder - multiple inputs.conf stanzas on the same folder

Stokers_23 — Tue, 21 Aug 2018 15:43:47 GMT

I'm attempting to configure my universal forwarder to read log files from a single directory with multiple subdirectories. We use log rotate so the files will be renamed with (1) up to (4) before starting again. I'm also trying to push those into the right index based on the file name. For example the top level directory is /srv/logs which has multiple subdirectories i.e

application
fileservice
proxyserver

each of these subdirectories contains multiple files from each environment (dev, int, prod etc) Here is an example file name.

application-prod.prod.log, i'm using the following inputs.conf which seems to work(ish). I've changed the monitor names to ensure they are treated as separate and i'm trying to blacklist anything I don't want to appear in each index.

[monitor:///srv/./logs]
blacklist = ppd..log$|prod..log$
sourcetype = service_log
index = nonprod
crcSalt =

[monitor:///srv/logs]
blacklist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = prod
crcSalt =

So in prod, I only want files that contain .prod and ppd, in nonprod I want devint, int, ft, infradev and nonprod.

So i'm wondering
- Are there better or more performant ways to configure these inputs
- Is there anyway I can check the data is correct in my indexes is correct (no prod data in non prod etc)
- If there are subdirectories should I be using recursive = true?
- The documentation says don't use crcSalt = with log rotate - however I see a number of initcrc errors - should I be setting a initcrclen = 2000 etc?

Sorry this is a long one, thanks for any help.

Thanks

Re: Universal forwarder - multiple inputs.conf stanzas on the same folder

gcusello — Tue, 21 Aug 2018 16:40:47 GMT

Hi Stokers_23,
for my mental order, i prefer to have different stanzas for each kind of files, so I'd use this stanzas:

[monitor:///srv/logs/*/*.devint.log]
sourcetype = service_log
index = nonprod
disabled = 0

[monitor:///srv/logs/*/*.int.log]
sourcetype = service_log
index = nonprod
disabled = 0

[monitor:///srv/logs/*/*.infradev.log]
sourcetype = service_log
index = nonprod
disabled = 0

[monitor:///srv/logs/*/*.nonprod.log]
sourcetype = service_log
index = nonprod
disabled = 0

[monitor:///srv/logs/*/*.prod.log]
sourcetype = service_log
index = prod
disabled = 0

[monitor:///srv/logs/*/*.ppd.log]
sourcetype = service_log
index = prod
disabled = 0

Otherwise, if you want to reduce the number of stanzas you could use whitelist instead black list

[monitor:///srv/./logs]
whitelist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = nonprod

[monitor:///srv/logs]
whitelist = ppd..log$|prod..log$
sourcetype = service_log
index = prod

Answering to your questions:

the more performant ways to configure these inputs is to create a stanza for each input with exact path;
to check if the data is correctly in your indexes, you can run a search like index=prod | stats count by source and verify the sources in your index, then repeat the same search for the nonprod index
If there are subdirectories you don't need to put recursive = true you can use asterisk (*) or three points (...)?
crcSalt = is used only when you want to take logs only from new files, it isn't your case?

I hope to be helpful for you.

Bye.
Giuseppe

Re: Universal forwarder - multiple inputs.conf stanzas on the same folder

Stokers_23 — Thu, 23 Aug 2018 15:41:35 GMT

Hey, thanks for the detailed answer..

I have one issue left you might be able to help with, using the whitelists works really well. There is only one issue - the files named 'nonprod' are sometimes appearing in the prod index. I've tried blacklisting nonprod in the bottom stanza but that doesnt seem to work.

The file names contain '-prod.prod' so I could try the below?

[monitor:///srv/./logs]
whitelist = devint..log$|int..log$|ft..log$|infradev..log$|nonprod.*.log$
sourcetype = service_log
index = nonprod

[monitor:///srv/logs]
whitelist = ppd..log$|-prod..log$
sourcetype = service_log
index = prod

Re: Universal forwarder - multiple inputs.conf stanzas on the same folder

gcusello — Thu, 23 Aug 2018 15:45:51 GMT

Hi Stokers_23,
yes the problem is that prod in contained in nonprod, so you have to be mo detailed in your regex.
Bye.
Giuseppe