Splunk Search

Sourcetype Override

iatwal
Path Finder

We have around 15 files we're ingesting into Splunk all of them have the same format:

//logs/TEST/mike/TEST1/syslog.log
//logs/PROD/julie/TEST1/auth.log
//logs/ACPT/rob/TEST1/mail.log
//logs/DEV/frank/TEST1/kern.log
//logs/STG/rick/TEST1/debug.log
//logs/STUFF/ken/TEST1/messages.log

We would like to change the sourcetype of the names as they come in. So sourectype should automatically set to:

mike
julie
rob
frank
rick
ken

Can this be done dynamically via inputs.conf? Or does this need to happen in props/transforms.conf? What would my regex look like? All help is appreciated...

0 Karma

hhGA
Communicator

Hi,

To do this dynamically you would use props.confs and transforms.conf using the following code:

props.conf:
TRANSFORMS-change_sourcetype = change_sourcetype

EDIT: Please excuse the below formatting; I couldn't get the line breaks to work.

transforms.conf:
[change_sourcetype]
REGEX = (?U)\/\/logs\/\S+\/(\S+)\/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY=source

Splunk will need to be restarted for the changes to take affect.

I've not tested it but it should work. Let me know if you have any issues.

Also, if they all have the same format then I'd recommend that they all share the same sourcetype. Have you considered using a different field? This would be done using a search time extraction.

0 Karma

iatwal
Path Finder

so I'm struggling with the REGEX, mind helping here as we're very weak in it.

Do you mind messing with this, we're looking to make the profile_api our sourcetype...

https://regex101.com/r/GYYFmL/5

0 Karma

hhGA
Communicator

(?U)\/\S+\/\S+\/(?<sourcetype>.+)\/ should work.

I noticed that your regex101 was set to python. Splunk uses pcre regex for extractions.

I have missed SOURCE_KEY=source from the transforms.conf in my original answer which has now been updated.

0 Karma

iatwal
Path Finder

Another question, what does

sourcetype::AIDX

The AIDX mean?

0 Karma

hhGA
Communicator

Apologies, this was left over from my original conf file. I have corrected the original post.

$1 (the correct value) references the first capture group of the regex.

0 Karma

iatwal
Path Finder

Another question, what does

sourcetype::AIDX

The AIDX mean?

0 Karma

iatwal
Path Finder

I forgot to state that these files sit on a rsyslog server (with a forwarder installed) and we ingest them in via inputs.conf. Will the props and transforms need to be on my indexers? I appreciate your response.

0 Karma

hhGA
Communicator

Hi Iatwal,

If the rsyslog server is using a universal forwarder then you will have to use the props.conf and transforms.conf on your indexer(s).

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...