Getting Data In

Applying changes to logs whose sourcetype has been changed on a per-event basis

ehowardl3
Path Finder

I have the need to change the sourcetype of certain logs on a per-event basis, then apply further changes on the new sourcetype in props.conf after the sourcetype change. For example:

transforms.conf:

 [set_new_sourcetype]
 DEST_KEY = MetaData:Sourcetype
 REGEX = some_regex
 FORMAT = sourcetype::my:new:sourcetype

props.conf:

[old:sourcetype]
TRANSFORMS-sourcetype_change = set_new_sourcetype

[my:new:sourcetype]
SEDCMD-whatever = s/change/that/g

However, the SEDCMD does not work on the new sourcetype. If I move the SEDCMD to the original sourcetype it works. Is there a workaround for this? If not, what's the point of being able to change the sourcetype on a per-event basis if you can't do anything with it afterwards?

Thanks

0 Karma

koshyk
Super Champion

Please have a try like

props.conf

[old:sourcetype]
TRANSFORMS-sourcetype_change = set_new_sourcetype,substitute_whatever

transforms.conf

[set_new_sourcetype]
 DEST_KEY = MetaData:Sourcetype
 REGEX = some_regex
 FORMAT = sourcetype::my:new:sourcetype

  [substitute_whatever]
  REGEX = (findSomeKey)
  FORMAT = substitutedValue
  DEST_KEY = _raw
0 Karma

ehowardl3
Path Finder

Hi @koshyk, thanks for the reply. I'm not sure this will work in my case. I'm setting the sourcetype based on one string that is only common to those logs, however the string I need removed from the new sourcetype is common to the logs in both the new and old sourcetypes. The way this is written, I believe it will affect both of my sourcetypes, correct?

0 Karma

koshyk
Super Champion

That is correct. So you essentially need "almost duplicate" data in two sourcetypes.
If that's the case, in the above logic, do as two lines. Ensure that the TRANSFORM names are increasing order of ASCII . Something like below

 [old:sourcetype]
 TRANSFORMS-1sourcetype_change = set_new_sourcetype
 TRANSFORMS-2substitute_whatever = substitute_whatever
0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...