Sourcetype Override

[iatwal]

We have around 15 files we're ingesting into Splunk all of them have the same format:

//logs/TEST/mike/TEST1/syslog.log
//logs/PROD/julie/TEST1/auth.log
//logs/ACPT/rob/TEST1/mail.log
//logs/DEV/frank/TEST1/kern.log
//logs/STG/rick/TEST1/debug.log
//logs/STUFF/ken/TEST1/messages.log

We would like to change the sourcetype of the names as they come in. So sourectype should automatically set to:

mike
julie
rob
frank
rick
ken

Can this be done dynamically via inputs.conf? Or does this need to happen in props/transforms.conf? What would my regex look like? All help is appreciated...

[hhGA]

Hi,

To do this dynamically you would use props.confs and transforms.conf using the following code:

props.conf:
TRANSFORMS-change_sourcetype = change_sourcetype

EDIT: Please excuse the below formatting; I couldn't get the line breaks to work.

transforms.conf:
[change_sourcetype]
REGEX = (?U)\/\/logs\/\S+\/(\S+)\/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY=source


Splunk will need to be restarted for the changes to take affect.

I've not tested it but it should work. Let me know if you have any issues.

Also, if they all have the same format then I'd recommend that they all share the same sourcetype. Have you considered using a different field? This would be done using a search time extraction.

[iatwal]

so I'm struggling with the REGEX, mind helping here as we're very weak in it.

Do you mind messing with this, we're looking to make the profile_api our sourcetype...

https://regex101.com/r/GYYFmL/5

[hhGA]

(?U)\/\S+\/\S+\/(?<sourcetype>.+)\/ should work.

I noticed that your regex101 was set to python. Splunk uses pcre regex for extractions.

I have missed SOURCE_KEY=source from the transforms.conf in my original answer which has now been updated.

[iatwal]

Another question, what does

sourcetype::AIDX

The AIDX mean?

[hhGA]

Apologies, this was left over from my original conf file. I have corrected the original post.

$1 (the correct value) references the first capture group of the regex.

[iatwal]

Another question, what does

sourcetype::AIDX

The AIDX mean?

[iatwal]

I forgot to state that these files sit on a rsyslog server (with a forwarder installed) and we ingest them in via inputs.conf. Will the props and transforms need to be on my indexers? I appreciate your response.

[hhGA]

Hi Iatwal,

If the rsyslog server is using a universal forwarder then you will have to use the props.conf and transforms.conf on your indexer(s).