Getting Data In

How to seperate different Sourcetype logs from single syslog IP source based on Regex

NightShark
Path Finder

Greetings,

I am trying to get different log types such as security and audit logs for example from a single IP source from my HF instance, how exactly should I be settings my settings in Inputs, Transforms and Props conf in my HF to accomplish this?

Thanks,

 

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @NightShark,

you have to override sourcetypes on event basis.

You can find a description of this job at https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides#:~:text=You%20c....

In few words, you have to identify the regexes for each destination sourcetype and then put in your Indexers or (if present) in your Heavy Forwarders:

props.conf

[your_sourcetype]
TRANSFORMS-override_sourcetype = override_sourcetype_1, override_sourcetype_2

transforms.conf

[override_sourcetype_1]
REGEX = <your_regex_1>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

[override_sourcetype_2]
REGEX = <your_regex_2>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

 In this example I use two destination sourcetypes.

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @NightShark,

you have to override sourcetypes on event basis.

You can find a description of this job at https://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides#:~:text=You%20c....

In few words, you have to identify the regexes for each destination sourcetype and then put in your Indexers or (if present) in your Heavy Forwarders:

props.conf

[your_sourcetype]
TRANSFORMS-override_sourcetype = override_sourcetype_1, override_sourcetype_2

transforms.conf

[override_sourcetype_1]
REGEX = <your_regex_1>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

[override_sourcetype_2]
REGEX = <your_regex_2>
FORMAT = sourcetype::<your_custom_sourcetype_value>
DEST_KEY = MetaData:Sourcetype

 In this example I use two destination sourcetypes.

Ciao.

Giuseppe

youngsuh
Contributor

@gcusello 

Could you review mind?  Here is my inputs.conf

[monitor:///var/log/$mask_host1$/syslog]
disabled = false
#initCrcLength = 800
crcSalt = <SOURCE>
index = test
sourcetype = uag:syslog

[monitor:///var/log/$mask_host2$/syslog]
disabled = false
#initCrcLength = 800
crcSalt = <SOURCE>
index = test
sourcetype = uag:syslog

[monitor:///var/log/$mask_host3/syslog]
disabled = false
#initCrcLength = 800
crcSalt = <SOURCE>
index = test
sourcetype = uag:syslog

Here is props.conf on the HF.

[uag:syslog]
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

Here is my transform.conf on the HF.

[uag:syslog]
REGEX = :\d\d\s+\w{5}\w{4}\suag-admin\:
FORMAT = sourcetype::vmware:uag:admin
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

[uag:syslog]
REGEX = :\d\d\s+\w{5}\w{4}\suag-audit\:
FORMAT = sourcetype::vmware:uag:admin
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

[uag:syslog]
REGEX = :\d\d\s+\w{5}\w{4}\suag-esmanager\:
FORMAT = sourcetype::vmware:uag:esmanager
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

Is there anything I am doing wrong in the configuration?

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @youngsuh,

please don't attach a new question to an answered one: creating a new one you'll have faster and probably better answers!

Anyway, what do you want to match with your regex: the source or the log content?

from your props.conf it seems that you want to match with the source field, but in this case the regex is wrong.

If you want to match with the content of the log delete the "SOURCE_KEY" row.

Ciao.

Giuseppe

0 Karma

youngsuh
Contributor

Will create a new post next time.  I want to match log content.  So, I delete 

SOURCE_KEY = MetaData:Source

you meant transform.conf right?  not props.conf

The props.conf should be change to match sourcetype right?

TRANSFORMS-uag:syslog = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

 

0 Karma

NightShark
Path Finder

Greetings,

Thank you very much gcusello,

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...