topic Re: How to seperate different Sourcetype logs from single syslog IP source based on Regex in Getting Data In

How to seperate different Sourcetype logs from single syslog IP source based on Regex

NightShark — Wed, 16 Feb 2022 09:46:20 GMT

Greetings,

I am trying to get different log types such as security and audit logs for example from a single IP source from my HF instance, how exactly should I be settings my settings in Inputs, Transforms and Props conf in my HF to accomplish this?

Thanks,

Re: How to seperate different Sourcetype logs from single syslog IP source based on Regex

gcusello — Wed, 16 Feb 2022 10:25:34 GMT

Hi @NightShark,

you have to override sourcetypes on event basis.

You can find a description of this job at https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides#:~:text=You%20can%20override%20source%20types,directly%20on%20the%20instance%20itself.

In few words, you have to identify the regexes for each destination sourcetype and then put in your Indexers or (if present) in your Heavy Forwarders:

props.conf

[your_sourcetype] TRANSFORMS-override_sourcetype = override_sourcetype_1, override_sourcetype_2

transforms.conf

[override_sourcetype_1] REGEX = <your_regex_1> FORMAT = sourcetype::<your_custom_sourcetype_value> DEST_KEY = MetaData:Sourcetype [override_sourcetype_2] REGEX = <your_regex_2> FORMAT = sourcetype::<your_custom_sourcetype_value> DEST_KEY = MetaData:Sourcetype

In this example I use two destination sourcetypes.

Ciao.

Giuseppe

Re: How to seperate different Sourcetype logs from single syslog IP source based on Regex

NightShark — Wed, 16 Feb 2022 10:46:50 GMT

Greetings,

Thank you very much gcusello,

Re: How to seperate different Sourcetype logs from single syslog IP source based on Regex

youngsuh — Mon, 19 Sep 2022 14:45:18 GMT

@gcusello

Could you review mind? Here is my inputs.conf

[monitor:///var/log/$mask_host1$/syslog] disabled = false #initCrcLength = 800 crcSalt = <SOURCE> index = test sourcetype = uag:syslog [monitor:///var/log/$mask_host2$/syslog] disabled = false #initCrcLength = 800 crcSalt = <SOURCE> index = test sourcetype = uag:syslog [monitor:///var/log/$mask_host3/syslog] disabled = false #initCrcLength = 800 crcSalt = <SOURCE> index = test sourcetype = uag:syslog

Here is props.conf on the HF.

[uag:syslog] NO_BINARY_CHECK = true category = Custom pulldown_type = 1 TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

Here is my transform.conf on the HF.

[uag:syslog] REGEX = :\d\d\s+\w{5}\w{4}\suag-admin\: FORMAT = sourcetype::vmware:uag:admin SOURCE_KEY = MetaData:Source DEST_KEY = MetaData:Sourcetype [uag:syslog] REGEX = :\d\d\s+\w{5}\w{4}\suag-audit\: FORMAT = sourcetype::vmware:uag:admin SOURCE_KEY = MetaData:Source DEST_KEY = MetaData:Sourcetype [uag:syslog] REGEX = :\d\d\s+\w{5}\w{4}\suag-esmanager\: FORMAT = sourcetype::vmware:uag:esmanager SOURCE_KEY = MetaData:Source DEST_KEY = MetaData:Sourcetype

Is there anything I am doing wrong in the configuration?

Re: How to seperate different Sourcetype logs from single syslog IP source based on Regex

gcusello — Tue, 20 Sep 2022 06:28:25 GMT

Hi @youngsuh,

please don't attach a new question to an answered one: creating a new one you'll have faster and probably better answers!

Anyway, what do you want to match with your regex: the source or the log content?

from your props.conf it seems that you want to match with the source field, but in this case the regex is wrong.

If you want to match with the content of the log delete the "SOURCE_KEY" row.

Ciao.

Giuseppe

Re: How to seperate different Sourcetype logs from single syslog IP source based on Regex

youngsuh — Tue, 20 Sep 2022 16:39:38 GMT

Will create a new post next time. I want to match log content. So, I delete

SOURCE_KEY = MetaData:Source

you meant transform.conf right? not props.conf

The props.conf should be change to match sourcetype right?

TRANSFORMS-uag:syslog = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager