I'm having a trouble splitting syslog data coming in over UDP:514 to their own index and transforming the respective sourcetype and source for the data. I'm working with Barracuda and CiscoASA & CiscoESA data that is being sent over UDP 514. I've tested the following examples but for some reason only the Cisco ASA is showing up in my search head and saving to Barracuda... I'm all mixed up
[override_index_cisco] FORMAT = cisco DEST_KEY = _MetaData:Index [override_index_barracuda] FORMAT = barracuda DEST_KEY = _MetaData:Index [override_sourcetype_cisco_asa_st] SOURCE_KEY = MetaData:Host FORMAT = sourcetype::cisco_asa DEST_KEY = MetaData:Sourcetype [override_sourcetype_cisco_esa_st] SOURCE_KEY = MetaData:Host FORMAT = sourcetype::cisco_esa DEST_KEY = MetaData:Sourcetype [override_sourcetype_barracuda] SOURCE_KEY = MetaData:Host FORMAT = sourcetype::barracuda DEST_KEY = MetaData:Sourcetype [override_source_cisco_asa_st] FORMAT = source::cisco_asa DEST_KEY = MetaData:Source [override_source_cisco_esa_st] FORMAT = source::cisco_esa DEST_KEY = MetaData:Source [override_source_barracuda] FORMAT = source::barracuda DEST_KEY = MetaData:Source
[source::udp:514] TRANSFORMS-changesourcetype = cisco_esa_st, cisco_asa_st, barracuda TRANSFORMS-index = cisco, barracuda
I think you may be a bit confused on how this all works.
Your props looks almost good enough. But to clarify, this is where we start the process. Data is going to come in on port 514 and hit that stanza in your props.
Each comma-separated value in your TRANSFORMS settings should correspond to a stanza in the transforms.conf. Yours don't seem to. But the way you have the named in your transforms looks good, so just match those in props.
Splunk will call each of those stanzas in transforms in the order you have specified. And if you have multiple TRANSFORMS settings (like you do in this case), they are applied in lexicographic order (I think). So in your example, splunk will call of the stanzas in the TRANSFORMS-changesourcetype settings first and then all of the stanzas in the TRANSFORMS-index setting.
in your transforms.conf, the key concept I think you're missing is the REGEX setting. You are just blindly setting all of those fields without checking to see if you should be. The regex would get applied to the source key and the dest key would only get updated if it was a match.
At a high level, this is the concept.
this is what you're going to apply the regex too. It defaults to _raw, so if you don't specify this, the regex will be applied to the entire event. So depending how you want to decide where this event came from, you need to choose the source key accordingly.
Also, if you specify all of the sourcetypes stanzas first in props settings, you can use the sourcetype as the source key to determine the source/index, because it will be overridden already.
the regex to match against. The format/dest_key portion of the stanza will only apply if the regex specified here matches the source key
If the regex matches the source key, this is the field that you're going to change (at a glance, this looks fine in your example)
assuming the regex matched, this is the format to apply the dest_key (at a glance, this looks fine in your example)
So ultimately, you need to determine how to identify events from each source. Then using the source_key and regex settings, make sure each stanza only matches the events it should.
Hope that helps a bit. It's not exactly an example like you asked but thought it might be better to try to explain. These types of example are all over the place out here...and I think some are in the docs too.
I'm still testing this and I'm trying to understand how all this connects, would you mind running through an example.
my inputs.conf under /system/local/
[udp://514] connection_host = dns index = index_syslog sourcetype = syslog
Props.conf <---- I'm not sure how to combine the two transforms-index stanzas into one so that I can just list the indexes I want to transform separated by comma. Does the stanza name need to match my inputs stanza name??
[override_index_barracuda] TRANSFORMS-index = barracuda [override_index_cisco] TRANSFORMS-index = cisco
Transforms.conf <<--- the regex technically works but could be refined. I also wouldn't mind combining the two stanzas if possible.
[override_index_barracuda] REGEX = (?<![\w\d])barracuda(?![\w\d]) FORMAT = barracuda DEST_KEY = _MetaData:Index [override_index_cisco] REGEX = (?<![\w\d])cisco(?![\w\d]) FORMAT = cisco DEST_KEY = _MetaData:Index
I think your inputs and transforms look ok (assuming the regex works), but maybe some help with props yet.
So the stanza in props needs to be the sourcetype (or source or host) of the data coming in. And then values for the TRANSFORMS settings need to match stanzas in transforms.conf. So maybe something like this would work here?
because the data is going to have a sourcetype of syslog, that's the stanza i use here. And the values on the right hand side of the transforms settings will match the stanza in transforms.
[syslog] TRANSFORMS-override-indexes = override_index_barracuda, override_index_cisco
So now syslog data should call both of those "override" stanzas you have defined in transforms. And if the regex matches the event, then it should update the index accordingly