Getting Data In

How to seperate different Sourcetype logs from single syslog IP source based on Regex

NightShark
Path Finder

Greetings,

I am trying to get different log types such as security and audit logs for example from a single IP source from my HF instance, how exactly should I be settings my settings in Inputs, Transforms and Props conf in my HF to accomplish this?

Thanks,

 

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @NightShark,

you have to override sourcetypes on event basis.

You can find a description of this job at https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides#:~:text=You%20c....

In few words, you have to identify the regexes for each destination sourcetype and then put in your Indexers or (if present) in your Heavy Forwarders:

props.conf

[your_sourcetype]
TRANSFORMS-override_sourcetype = override_sourcetype_1, override_sourcetype_2

transforms.conf

[override_sourcetype_1]
REGEX = <your_regex_1>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

[override_sourcetype_2]
REGEX = <your_regex_2>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

 In this example I use two destination sourcetypes.

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @NightShark,

you have to override sourcetypes on event basis.

You can find a description of this job at https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides#:~:text=You%20c....

In few words, you have to identify the regexes for each destination sourcetype and then put in your Indexers or (if present) in your Heavy Forwarders:

props.conf

[your_sourcetype]
TRANSFORMS-override_sourcetype = override_sourcetype_1, override_sourcetype_2

transforms.conf

[override_sourcetype_1]
REGEX = <your_regex_1>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

[override_sourcetype_2]
REGEX = <your_regex_2>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

 In this example I use two destination sourcetypes.

Ciao.

Giuseppe

youngsuh
Contributor

@gcusello 

Could you review mind?  Here is my inputs.conf

[monitor:///var/log/$mask_host1$/syslog]
disabled = false
#initCrcLength = 800
crcSalt = <SOURCE>
index = test
sourcetype = uag:syslog

[monitor:///var/log/$mask_host2$/syslog]
disabled = false
#initCrcLength = 800
crcSalt = <SOURCE>
index = test
sourcetype = uag:syslog

[monitor:///var/log/$mask_host3/syslog]
disabled = false
#initCrcLength = 800
crcSalt = <SOURCE>
index = test
sourcetype = uag:syslog

Here is props.conf on the HF.

[uag:syslog]
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

Here is my transform.conf on the HF.

[uag:syslog]
REGEX = :\d\d\s+\w{5}\w{4}\suag-admin\:
FORMAT = sourcetype::vmware:uag:admin
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

[uag:syslog]
REGEX = :\d\d\s+\w{5}\w{4}\suag-audit\:
FORMAT = sourcetype::vmware:uag:admin
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

[uag:syslog]
REGEX = :\d\d\s+\w{5}\w{4}\suag-esmanager\:
FORMAT = sourcetype::vmware:uag:esmanager
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

Is there anything I am doing wrong in the configuration?

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

please don't attach a new question to an answered one: creating a new one you'll have faster and probably better answers!

Anyway, what do you want to match with your regex: the source or the log content?

from your props.conf it seems that you want to match with the source field, but in this case the regex is wrong.

If you want to match with the content of the log delete the "SOURCE_KEY" row.

Ciao.

Giuseppe

0 Karma

youngsuh
Contributor

Will create a new post next time.  I want to match log content.  So, I delete 

SOURCE_KEY = MetaData:Source

you meant transform.conf right?  not props.conf

The props.conf should be change to match sourcetype right?

TRANSFORMS-uag:syslog = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

 

0 Karma

NightShark
Path Finder

Greetings,

Thank you very much gcusello,

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...