How to change sourcetype at index time based on the logs content?

[GaetanVP]

Hello Splunkers,

I have a question, would it be possible to assign a specific sourcetype to some logs inside a input stanza, depending on the content of the log itself (based on the key / fields extracted or some regex...).

For instance :

[monitor:///whatever]
if foo = bar
  sourcetype = scr_type_1
else 
  sourcetype = scr_type_1

I have few hope about this one...

Thanks a lot,
GaetanVP

[isoutamo]

Hi

At least I don't know any option how you can do it in inputs.conf.

But you could do it on HF/IDX with props.conf and transform.conf and clonesourcetype.

r. Ismo

[GaetanVP]

Hello @isoutamo,

Yes but with my props.conf file, I need to specify a sourcetype (or a host, or source, or rule...)  in the [<spec>] stanza, based on this page : https://docs.splunk.com/Documentation/Splunk/9.0.5/Admin/Propsconf

The thing is, I want to clone only a part of the logs which sourcetype is "sourcetypeA" and I do not know how I can say "clone only a part of those logs base on key/value foo=bar (or a regex)".

Is it possible to clone only a part of the logs incoming based on certain rules ? That would be very helpful...

Thanks,

GaetanVP

[isoutamo]

You need to clone and also drop away other events from both sourcetypes. I think that @dural_yyz shows already how to do it on original sourcetype.

[dural_yyz · Accepted solution]

inputs.conf

[monitor:///whatever/file.txt]
sourcetype = src:type:original

props.conf

[src:type:original]
TRANSFORMS-sourcetype = overridesourcetype_original 

transforms.conf

[overridesourcetype_original]
DEST_KEY = _MetaData:Sourcetype
REGEX = .
FORMAT = sourcetype::src:type:new-value 

This answers your original question of how to set an override value of the original sourcetype.

I haven't tried to change the sourcetype based on a match of _raw but I suspect that it would be possible.  Looking over the spec sheet for transforms.conf if you set the SOURCE_KEY = _raw the regex will apply there but the FORMAT would apply to the DEST_KEY.

However, your latest comment about cloning an event concerns me because I don't know of any configuration that will duplicate the event into multiple sourcetypes.

[GaetanVP]

Hello @dural_yyz thanks for you answer !

My main purpose was to clone only a subset of events (based on regex).

From what I read, the CLONE_SOURCETYPE spec will automatically clone ALL EVENTS from the the sourcetype given (correct me if I'm wong).

So my strategy will be to use your answer to first override the sourcetype of a subset of original event (the events that will match the REGEX field) and then use the CLONE_SOURCETYPE only on the new sourcetype.

That should do the job...

Thanks for your time,

GaetanVP

[dural_yyz]

Interesting, from what I have just read this will require you to have three source types but keep an eye on the parsing queue.  It would be unhelpful if clone_sourcetype wants to occur before the rewrite of the meta field making this solution a non-option.  I've never tested so best of luck and I hope it works.

[GaetanVP]

Hello @dural_yyz, here are some updates !

So the above approach didn't work as expected... The thing is after I changed the sourcetype of my events, they will directly go to the indexing phase, I do not think it's possible to "tell them to go back to parsing queue" in order to apply a CLONE_SOURCETYPE afterwards...

So I tried something else to clone and forward (to another Splunk HF) only a subset of my events... (nasty but for now I have no other options...). The main idea is to change the _TCP_ROUTING value for the event I want to clone, those events will be selected with the REGEX and directly forward to the wanted destination. Afterwards I will clean the events that were not changed by my REGEX (with an "inverse" REGEX) by sending them to null queue.

Here is an example with a JSON incoming raw data (I assigned them the sourcetype "mysourcetype") :

{
  "Foo": "Bar",
  "Hello": "World"
},
{
  "Foo": "Bar",
  "Hello": "Again"
}

props.conf 

[mysourcetype]
TRANSFORMS-foo-clone = trans-clone

[mysourcetype:cloned]
TRANSFORMS-bar-drop = trans-drop

transforms.conf

# clone all events and change tcp output for specific events mathcing the regex
[trans-clone]
CLONE_SOURCETYPE = mysourcetype:cloned
REGEX = "Hello":\s*"World"
DEST_KEY = _TCP_ROUTING
FORMAT = my_specific_output

# drop duplicated and not forwarded logs
[trans-drop]
REGEX = (?s)^(?!.*"Workload"\s*:\s*"Aip").*$
DEST_KEY = queue
FORMAT = nullQueue

I tested it and do the job for me ! I do not like the fact to use two "opposite REGEX" because of resources usage and if your REGEX is not okay, you will end up with duplicated unwanted data.

Thanks you for your time and @isoutamo for the good hint about dropping some data  😋

GaetanVP