topic Re: Problems with sourcetypes on syslog collectors-How can I set a sourcetype using REGEX? in Getting Data In

Problems with sourcetypes on syslog collectors-How can I set a sourcetype using REGEX?

Fonzie2k — Thu, 13 Oct 2022 14:31:48 GMT

Hi,

I have multiple syslog collectors (practically a heavy forwarder that picks up logs from disk).
I am struggling to find a way of setting a specific sourcetype for parts of this logs that are picked up from disk.

/data/syslog/ contains thousands of folders with IP adresses, and i want to set a specific sourcetype for lets say 100 of them... Ive tried using regex and whitelist, but it seems like two stanzas with the same name wont work:

[monitor:///data/syslog/tcp/.../*.log] sourcetype = rsyslog host_segment = 4 index = xxx_syslog blacklist = .*\.gz$ [monitor:///data/syslog/tcp/.../*.log] sourcetype = vmw-syslog host_segment = 4 index = xxx_syslog blacklist = .*\.gz$ whitelist = \/data\/syslog\/tcp\/(10\.21[1289]\.75\.\d+|10\.143\.15\.\d+|10\.21[01]\.70\.\d+|10\.250\.191\.50|10\.30\.221\.19[1-2]|11\.36\.1[128]\.\d+|10\.37\.12\.\d+|10\.45\.[12]\.\d+|10\.6[23]\.12.\d+|10\.63\.10\.20|10\.67\.(0|64)\.\d+|10\.67\.67\.67)\/

Any idea on how i can set an sourcetype using REGEX?
(I can not rewrite the sourcetype on a heavy forwarder, because this data should be parsed and get a new sourcetype from an TA app (vmware esxilogs), and i cant parse data two times).

Re: Problems with sourcetypes on syslog collectors-How can I set a sourcetype using REGEX?

starcher — Fri, 14 Oct 2022 04:47:03 GMT

Use syslogs own configuration to put different types in different folders such as based on the sending system source ip. Then just set the sourcetype by path on that folder in your inputs. How you do that in syslog will vary on which one you are using.

Re: Problems with sourcetypes on syslog collectors-How can I set a sourcetype using REGEX?

Fonzie2k — Wed, 19 Oct 2022 09:35:22 GMT

Thanks for the suggestion. We rather not touch the rsyslog config, as we want to keep it standard across multiple customers/tenants. Our solution was to create the following transforms on the HF's, to rewrite the sourcetype from spesific hosts (regex):

props.conf: [rsyslog] TRANSFORMS-hemit_force_sourcetype_vmware1 = hemit_force_sourcetype_vmware1 transforms.conf: [hemit_force_sourcetype_vmware1] SOURCE_KEY = MetaData:Host REGEX = ^host::(10\.24[1289]\.70\.\d+|etc|etc) DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::vmw-syslog

Re: Problems with sourcetypes on syslog collectors-How can I set a sourcetype using REGEX?

FrankVl — Wed, 19 Oct 2022 14:01:48 GMT

But that solution means you can't apply sourcetype specific index-time config like timestamping and linebreaking, source/sourcetype overrides etc.. So that's far from ideal and will certainly cause a bunch of headaches with certain types of data?

Crafting your syslog config such that it puts data that needs a different sourcetype into separate folders is typically the way to go. That way you can easily write distinct input stanzas for each sourcetype.