Problems with sourcetypes on syslog collectors-How can I set a sourcetype using REGEX?

[Fonzie2k]

Hi,

I have multiple syslog collectors (practically a heavy forwarder that picks up logs from disk).
I am struggling to find a way of setting a specific sourcetype for parts of this logs that are picked up from disk.

/data/syslog/ contains thousands of folders with IP adresses, and i want to set a specific sourcetype for lets say 100 of them... Ive tried using regex and whitelist, but it seems like two stanzas with the same name wont work:

[monitor:///data/syslog/tcp/.../*.log]

sourcetype = rsyslog
host_segment = 4
index = xxx_syslog
blacklist = .*\.gz$

[monitor:///data/syslog/tcp/.../*.log]

sourcetype = vmw-syslog
host_segment = 4
index = xxx_syslog
blacklist = .*\.gz$
whitelist = \/data\/syslog\/tcp\/(10\.21[1289]\.75\.\d+|10\.143\.15\.\d+|10\.21[01]\.70\.\d+|10\.250\.191\.50|10\.30\.221\.19[1-2]|11\.36\.1[128]\.\d+|10\.37\.12\.\d+|10\.45\.[12]\.\d+|10\.6[23]\.12.\d+|10\.63\.10\.20|10\.67\.(0|64)\.\d+|10\.67\.67\.67)\/

Any idea on how i can set an sourcetype using REGEX?
(I can not rewrite the sourcetype on a heavy forwarder, because this data should be parsed and get a new sourcetype from an TA app (vmware esxilogs), and i cant parse data two times).

[Fonzie2k · Accepted solution]

Thanks for the suggestion. We rather not touch the rsyslog config, as we want to keep it standard across multiple customers/tenants. Our solution was to create the following transforms on the HF's, to rewrite the sourcetype from spesific hosts (regex):

props.conf:
[rsyslog]
TRANSFORMS-hemit_force_sourcetype_vmware1 = hemit_force_sourcetype_vmware1

transforms.conf:
[hemit_force_sourcetype_vmware1]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(10\.24[1289]\.70\.\d+|etc|etc)

DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::vmw-syslog

[FrankVl]

But that solution means you can't apply sourcetype specific index-time config like timestamping and linebreaking, source/sourcetype overrides etc.. So that's far from ideal and will certainly cause a bunch of headaches with certain types of data?

Crafting your syslog config such that it puts data that needs a different sourcetype into separate folders is typically the way to go. That way you can easily write distinct input stanzas for each sourcetype.

[starcher]

Use syslogs own configuration to put different types in different folders such as based on the sending system source ip. Then just set the sourcetype by path on that folder in your inputs. How you do that in syslog will vary on which one you are using.