topic Re: Inputs.conf help in Getting Data In

Inputs.conf help

plumainwfs — Tue, 29 Sep 2020 12:54:16 GMT

I am trying to onboard ingest about 30 different log type from a single Source (Linux Server)

Currently the logs are being written and zipped by rsyslog in a particular folder (structure below):
/mnt/log/files/YYYY/MM/DD/hostname/filename.log.gz

What would be the best way to ingest the different logtypes

I was thinking about the following:

[monitor:///mnt/log/files/////]
host_segment=7
whitelist=(.+\access.gz)
index=webserver
sourcetype=access:apache

Not sure why the asterisks are not showing up on the preview;
it should be /mnt/log/files/aster*/aster*/aster*/aster*/

I was also thinking to ingest all 30 log types I would need to create 30 different configs?

Re: Inputs.conf help

lguinn2 — Thu, 16 Feb 2017 14:44:11 GMT

If the different log types are going to different indexes, then yes, you will need 30 different configs. But if the only difference will be the sourcetype, you could do an override in props.conf. You still have to configure all the different configs, but it might be easier to do. Take a look at this:

inputs.conf
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webserver

props.conf
[source::/mnt/log/files/.../access.gz]
sourcetype=access:apache

[source::/mnt/log/files/.../otherthing.gz]
sourcetype=st_otherthing

etc.

Re: Inputs.conf help

plumainwfs — Thu, 16 Feb 2017 15:47:28 GMT

It would most probably be about 4-5 different indexes
Best way to do this would be with the inputs as so..
inputs.conf
[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webserver

[monitor:///mnt/log/files/]
whitelist=.gz$
host_segment=7
index=webapp

All 30 log types would be defined as separate sourcetype
so best way do to this would be with the props.conf as you outlined?
props.conf
[source::/mnt/log/files/.../access.gz]
sourcetype=access:apache

[source::/mnt/log/files/.../error.gz]
sourcetype=st_otherthing

Re: Inputs.conf help

plumainwfs — Thu, 16 Feb 2017 18:28:45 GMT

Also what happens to a gz file that I am not aware of? will that get indexed? and will not have a pre-defined sourcetype?