topic Re: Applying changes to logs whose sourcetype has been changed on a per-event basis in Getting Data In

Applying changes to logs whose sourcetype has been changed on a per-event basis

ehowardl3 — Wed, 08 May 2019 14:12:49 GMT

I have the need to change the sourcetype of certain logs on a per-event basis, then apply further changes on the new sourcetype in props.conf after the sourcetype change. For example:

transforms.conf:

 [set_new_sourcetype]
 DEST_KEY = MetaData:Sourcetype
 REGEX = some_regex
 FORMAT = sourcetype::my:new:sourcetype

props.conf:

[old:sourcetype]
TRANSFORMS-sourcetype_change = set_new_sourcetype

[my:new:sourcetype]
SEDCMD-whatever = s/change/that/g

However, the SEDCMD does not work on the new sourcetype. If I move the SEDCMD to the original sourcetype it works. Is there a workaround for this? If not, what's the point of being able to change the sourcetype on a per-event basis if you can't do anything with it afterwards?

Thanks

Re: Applying changes to logs whose sourcetype has been changed on a per-event basis

koshyk — Wed, 08 May 2019 15:02:52 GMT

Please have a try like

props.conf

[old:sourcetype]
TRANSFORMS-sourcetype_change = set_new_sourcetype,substitute_whatever

transforms.conf

[set_new_sourcetype]
 DEST_KEY = MetaData:Sourcetype
 REGEX = some_regex
 FORMAT = sourcetype::my:new:sourcetype

  [substitute_whatever]
  REGEX = (findSomeKey)
  FORMAT = substitutedValue
  DEST_KEY = _raw

Re: Applying changes to logs whose sourcetype has been changed on a per-event basis

ehowardl3 — Wed, 08 May 2019 17:25:11 GMT

Hi @koshyk, thanks for the reply. I'm not sure this will work in my case. I'm setting the sourcetype based on one string that is only common to those logs, however the string I need removed from the new sourcetype is common to the logs in both the new and old sourcetypes. The way this is written, I believe it will affect both of my sourcetypes, correct?

Re: Applying changes to logs whose sourcetype has been changed on a per-event basis

koshyk — Wed, 08 May 2019 18:48:32 GMT

That is correct. So you essentially need "almost duplicate" data in two sourcetypes.
If that's the case, in the above logic, do as two lines. Ensure that the TRANSFORM names are increasing order of ASCII . Something like below

 [old:sourcetype]
 TRANSFORMS-1sourcetype_change = set_new_sourcetype
 TRANSFORMS-2substitute_whatever = substitute_whatever