Looking for examples for splitting syslog data to separate indexes and transforming sourcetype

[johnward4]

I'm having a trouble splitting syslog data coming in over UDP:514 to their own index and transforming the respective sourcetype and source for the data. I'm working with Barracuda and Cisco_ASA & Cisco_ESA data that is being sent over UDP 514. I've tested the following examples but for some reason only the Cisco ASA is showing up in my search head and saving to Barracuda... I'm all mixed up

Transforms.conf

[override_index_cisco]
FORMAT = cisco
DEST_KEY = _MetaData:Index

[override_index_barracuda]
FORMAT = barracuda
DEST_KEY = _MetaData:Index

[override_sourcetype_cisco_asa_st] 
SOURCE_KEY = MetaData:Host 
FORMAT = sourcetype::cisco_asa 
DEST_KEY = MetaData:Sourcetype

[override_sourcetype_cisco_esa_st] 
SOURCE_KEY = MetaData:Host 
FORMAT = sourcetype::cisco_esa 
DEST_KEY = MetaData:Sourcetype

[override_sourcetype_barracuda] 
SOURCE_KEY = MetaData:Host 
FORMAT = sourcetype::barracuda 
DEST_KEY = MetaData:Sourcetype

[override_source_cisco_asa_st]
FORMAT = source::cisco_asa
DEST_KEY = MetaData:Source

[override_source_cisco_esa_st]
FORMAT = source::cisco_esa
DEST_KEY = MetaData:Source

[override_source_barracuda]
FORMAT = source::barracuda
DEST_KEY = MetaData:Source

Props.conf

[source::udp:514] 
TRANSFORMS-changesourcetype = cisco_esa_st, cisco_asa_st, barracuda
TRANSFORMS-index = cisco, barracuda

[maciep · Accepted solution]

I think you may be a bit confused on how this all works.

props.conf
Your props looks almost good enough. But to clarify, this is where we start the process. Data is going to come in on port 514 and hit that stanza in your props.

Each comma-separated value in your TRANSFORMS settings should correspond to a stanza in the transforms.conf. Yours don't seem to. But the way you have the named in your transforms looks good, so just match those in props.

Splunk will call each of those stanzas in transforms in the order you have specified. And if you have multiple TRANSFORMS settings (like you do in this case), they are applied in lexicographic order (I think). So in your example, splunk will call of the stanzas in the TRANSFORMS-changesourcetype settings first and then all of the stanzas in the TRANSFORMS-index setting.

transforms.conf
in your transforms.conf, the key concept I think you're missing is the REGEX setting. You are just blindly setting all of those fields without checking to see if you should be. The regex would get applied to the source key and the dest key would only get updated if it was a match.

At a high level, this is the concept.

SOURCE_KEY:
this is what you're going to apply the regex too. It defaults to _raw, so if you don't specify this, the regex will be applied to the entire event. So depending how you want to decide where this event came from, you need to choose the source key accordingly.

Also, if you specify all of the sourcetypes stanzas first in props settings, you can use the sourcetype as the source key to determine the source/index, because it will be overridden already.

REGEX:
the regex to match against. The format/dest_key portion of the stanza will only apply if the regex specified here matches the source key

DEST_KEY:
If the regex matches the source key, this is the field that you're going to change (at a glance, this looks fine in your example)

FORMAT:
assuming the regex matched, this is the format to apply the dest_key (at a glance, this looks fine in your example)

So ultimately, you need to determine how to identify events from each source. Then using the source_key and regex settings, make sure each stanza only matches the events it should.

Hope that helps a bit. It's not exactly an example like you asked but thought it might be better to try to explain. These types of example are all over the place out here...and I think some are in the docs too.

[johnward4]

Awesome info @maciep ! thank you

[johnward4]

@maciep

I'm still testing this and I'm trying to understand how all this connects, would you mind running through an example.

my inputs.conf under /system/local/

[udp://514]
connection_host = dns
index = index_syslog
sourcetype = syslog

Props.conf <---- I'm not sure how to combine the two transforms-index stanzas into one so that I can just list the indexes I want to transform separated by comma. Does the stanza name need to match my inputs stanza name??

[override_index_barracuda]
TRANSFORMS-index = barracuda

[override_index_cisco]
TRANSFORMS-index = cisco

Transforms.conf <<--- the regex technically works but could be refined. I also wouldn't mind combining the two stanzas if possible.

[override_index_barracuda]
REGEX = (?<![\w\d])barracuda(?![\w\d])
FORMAT = barracuda
DEST_KEY = _MetaData:Index

[override_index_cisco]
REGEX = (?<![\w\d])cisco(?![\w\d])
FORMAT = cisco
DEST_KEY = _MetaData:Index

[maciep]

I think your inputs and transforms look ok (assuming the regex works), but maybe some help with props yet.

So the stanza in props needs to be the sourcetype (or source or host) of the data coming in. And then values for the TRANSFORMS settings need to match stanzas in transforms.conf. So maybe something like this would work here?

props
because the data is going to have a sourcetype of syslog, that's the stanza i use here. And the values on the right hand side of the transforms settings will match the stanza in transforms.

[syslog]
TRANSFORMS-override-indexes = override_index_barracuda, override_index_cisco

So now syslog data should call both of those "override" stanzas you have defined in transforms. And if the regex matches the event, then it should update the index accordingly