Archive

Sourcetype Override

Path Finder

We have around 15 files we're ingesting into Splunk all of them have the same format:

//logs/TEST/mike/TEST1/syslog.log
//logs/PROD/julie/TEST1/auth.log
//logs/ACPT/rob/TEST1/mail.log
//logs/DEV/frank/TEST1/kern.log
//logs/STG/rick/TEST1/debug.log
//logs/STUFF/ken/TEST1/messages.log

We would like to change the sourcetype of the names as they come in. So sourectype should automatically set to:

mike
julie
rob
frank
rick
ken

Can this be done dynamically via inputs.conf? Or does this need to happen in props/transforms.conf? What would my regex look like? All help is appreciated...

0 Karma

Communicator

Hi,

To do this dynamically you would use props.confs and transforms.conf using the following code:

props.conf:
TRANSFORMS-change_sourcetype = change_sourcetype

EDIT: Please excuse the below formatting; I couldn't get the line breaks to work.

transforms.conf:
[change_sourcetype]
REGEX = (?U)\/\/logs\/\S+\/(\S+)\/
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype
SOURCE_KEY=source

Splunk will need to be restarted for the changes to take affect.

I've not tested it but it should work. Let me know if you have any issues.

Also, if they all have the same format then I'd recommend that they all share the same sourcetype. Have you considered using a different field? This would be done using a search time extraction.

0 Karma

Path Finder

so I'm struggling with the REGEX, mind helping here as we're very weak in it.

Do you mind messing with this, we're looking to make the profile_api our sourcetype...

https://regex101.com/r/GYYFmL/5

0 Karma

Communicator

(?U)\/\S+\/\S+\/(?<sourcetype>.+)\/ should work.

I noticed that your regex101 was set to python. Splunk uses pcre regex for extractions.

I have missed SOURCE_KEY=source from the transforms.conf in my original answer which has now been updated.

0 Karma

Path Finder

Another question, what does

sourcetype::AIDX

The AIDX mean?

0 Karma

Communicator

Apologies, this was left over from my original conf file. I have corrected the original post.

$1 (the correct value) references the first capture group of the regex.

0 Karma

Path Finder

Another question, what does

sourcetype::AIDX

The AIDX mean?

0 Karma

Path Finder

I forgot to state that these files sit on a rsyslog server (with a forwarder installed) and we ingest them in via inputs.conf. Will the props and transforms need to be on my indexers? I appreciate your response.

0 Karma

Communicator

Hi Iatwal,

If the rsyslog server is using a universal forwarder then you will have to use the props.conf and transforms.conf on your indexer(s).

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!