topic Re: Sourcetype Override in Splunk Search

Sourcetype Override

iatwal — Thu, 27 Apr 2017 14:56:28 GMT

We have around 15 files we're ingesting into Splunk all of them have the same format:

//logs/TEST/mike/TEST1/syslog.log
//logs/PROD/julie/TEST1/auth.log
//logs/ACPT/rob/TEST1/mail.log
//logs/DEV/frank/TEST1/kern.log
//logs/STG/rick/TEST1/debug.log
//logs/STUFF/ken/TEST1/messages.log

We would like to change the sourcetype of the names as they come in. So sourectype should automatically set to:

mike
julie
rob
frank
rick
ken

Can this be done dynamically via inputs.conf? Or does this need to happen in props/transforms.conf? What would my regex look like? All help is appreciated...

Re: Sourcetype Override

hhGA — Thu, 27 Apr 2017 15:28:47 GMT

Hi,

To do this dynamically you would use props.confs and transforms.conf using the following code:

props.conf:
TRANSFORMS-change_sourcetype = change_sourcetype

EDIT: Please excuse the below formatting; I couldn't get the line breaks to work.

transforms.conf:
[change_sourcetype] REGEX = (?U)\/\/logs\/\S+\/(\S+)\/ FORMAT = sourcetype::$1 DEST_KEY = MetaData:Sourcetype SOURCE_KEY=source

Splunk will need to be restarted for the changes to take affect.

I've not tested it but it should work. Let me know if you have any issues.

Also, if they all have the same format then I'd recommend that they all share the same sourcetype. Have you considered using a different field? This would be done using a search time extraction.

Re: Sourcetype Override

iatwal — Thu, 27 Apr 2017 15:59:25 GMT

I forgot to state that these files sit on a rsyslog server (with a forwarder installed) and we ingest them in via inputs.conf. Will the props and transforms need to be on my indexers? I appreciate your response.

Re: Sourcetype Override

hhGA — Thu, 27 Apr 2017 16:03:51 GMT

Hi Iatwal,

If the rsyslog server is using a universal forwarder then you will have to use the props.conf and transforms.conf on your indexer(s).

Re: Sourcetype Override

iatwal — Thu, 27 Apr 2017 16:25:16 GMT

Another question, what does

sourcetype::AIDX

The AIDX mean?

Re: Sourcetype Override

iatwal — Thu, 27 Apr 2017 16:26:45 GMT

Another question, what does

sourcetype::AIDX

The AIDX mean?

Re: Sourcetype Override

hhGA — Thu, 27 Apr 2017 17:05:46 GMT

Apologies, this was left over from my original conf file. I have corrected the original post.

$1 (the correct value) references the first capture group of the regex.

Re: Sourcetype Override

iatwal — Thu, 27 Apr 2017 17:10:58 GMT

so I'm struggling with the REGEX, mind helping here as we're very weak in it.

Do you mind messing with this, we're looking to make the profile_api our sourcetype...

https://regex101.com/r/GYYFmL/5

Re: Sourcetype Override

hhGA — Thu, 27 Apr 2017 17:35:55 GMT

(?U)\/\S+\/\S+\/(?<sourcetype>.+)\/ should work.

I noticed that your regex101 was set to python. Splunk uses pcre regex for extractions.

I have missed SOURCE_KEY=source from the transforms.conf in my original answer which has now been updated.