topic Override Sourcetype in Splunk Search

Override Sourcetype

Bill_B — Mon, 28 Sep 2020 16:02:26 GMT

I'm trying to do a sourcetype override and not having much luck. I am trying to change the sourcetype from 2 hosts, from udp:514 to "PAN". I went through the "Override Sourcetype" guide and followed the example. When opening the transforms.conf and props.conf file there is a message that says, "Do not edit this file. Make edits in etc/system/local." Ok, so in "etc/system/local" I created tthe transforms.conf and props.conf files and placed the following inside:
"transforms.conf"

[set_sourcetype_syslog_pan]

REGEX = :\d\d\s+(?:\d+\s+|(?:user|daemon|local.?).\w+\s+)[?(10.0.1.21|10.0.1.23)[\w.-]]?\s

FORMAT = sourcetype::PAN

DEST_KEY = MetaData:Sourcetype

"props.conf"

[source::udp:514]

TRANSFORMS-changesourcetype = set_sourcetype_syslog_pan

This appears to of had no effect. All sourcetypes still show as "syslog". When I tried to test the REGEX in Splunk with the rex command I received the error: "Error in 'SearchParser': Missing a search command before '('.

So what's wrong here? The REGEX is right out of the guide, so I don't think it should error out.

Thank you.

Re: Override Sourcetype

lguinn2 — Wed, 05 Mar 2014 06:12:19 GMT

Data can only be transformed as it is being parsed - once it is written to the index, the transforms.conf settings have no effect.

You did the right thing by putting creating the files in the local directory. Without looking too closely, the syntax is right. But none of that is going to effect existing data. And I think you are doing this the hard way.

So, you need to do two things:

1) Set the sourcetype for incoming data. You can do this in inputs.conf for all data in an input like this:

[udp://514]
sourcetype=PAN

2) setting an alias for the sourcetype for existing data. You can do this under the "Settings" ("Manager" in older versions). Look for "Fields" and within that, choose the sourcetype alias. After the erroneous data ages out of the index, you can remove the alias.

Re: Override Sourcetype

Bill_B — Thu, 06 Mar 2014 01:57:58 GMT

Thanks for your answer lguinn, but I am trying to do a sourcetype override based on host. As in the example at the bottom of this doc: http://docs.splunk.com/Documentation/Splunk/6.0/Data/Advancedsourcetypeoverrides
I have multiple hosts sending events to udp:514 but I would like the events from certain hosts to be overridden and labeled differently.
Any ideas?

Re: Override Sourcetype

lguinn2 — Thu, 06 Mar 2014 23:18:19 GMT

New answer. I would do it like this:

transforms.conf

[set_sourcetype_syslog_pan]
REGEX = \d{2}:\d{2}:\d{2}(?:\.d+)\s+(?:10.0.1.21|10.0.1.23)
FORMAT = sourcetype::PAN
DEST_KEY = MetaData:Sourcetype

props.conf

[source::udp:514]
TRANSFORMS-cst = set_sourcetype_syslog_pan

Additional thoughts: syslog comes in a variety of formats. A couple of lines from your syslog data (anonymized of course) would help. I wouldn't assume that the REGEX from the manual would work in your specific case. Notice that I generalized it - a lot. I assumed that the host (ip address) came immediately after a time stamp, which is a syslog standard, and the way my syslog data appears.

Second, are you sure that the IP addresses (10.0.1.21 and 10.0.1.23) appear in the data?

Third, is the source name exactly udp:415?

Finally (I know I am repeating myself) - be sure that you are looking at new data after you make changes to props.conf and transforms.conf. Parsing changes are NOT retroactive; the old data will not be changed.

Re: Override Sourcetype

Bill_B — Thu, 06 Mar 2014 23:26:02 GMT

Thanks again. I'll give it a try.