Change a sourcetype at the indexer

[ITSplunk117]

Hello,

I'm to try changing the sourcetype at the indexer level based on the source.  First question is that possible on an indexer.  

Second would it work with props.conf referencing the transforms

transforms.conf

testchange

REGEX = .+

FORMAT = Sourcetype::testsourcetype

WRITE_META = true

thanks

[PickleRick]

There are two important things about this question.

1. As @livehybrid already pointed out, if you're getting your data as parsed (already processed by another "full" instance of Splunk Enterprise - typically a HF but your events might also originate at SH, DS, CM...) they will not be parsed again so your transforms will be "dead".

2. There is a question of what you're trying to achieve. If you overwrite your sourcetype (or source, or host) during ingestion, it will not change the ingestion pipeline - your event will still get treated as per the original sourcetype/source/host trio. Yes, in search-time it will be processed according to the new sourcetype, but the ingestion process is decided at the very beginning and is not altered "mid-flight" (with a possible exception of CLONE_SOURCETYPE but that is a relatively advanced topic).

[ITSplunk117]

Hello,

It did work if the data arrived from a UF.  Which as far as I can tell for this related ticket is how the data is coming in. 

What I was trying to achieve is changing a custom sourcetype to a standard one for a specific log type.  Since we do not have full access to specific splunk environments having it changed on the indexer level is likely the only way to do it.  

If there's a HF involved, then my team should have access to it and we can make the appropriate changes on the endpoint.  

[PickleRick]

OK. So as long as you're getting cooked data, not cooked and parsed, your transforms should work. Just - as I said - be aware that by rewriting sourcetype you're not changing the ingestion pipeline. Therefore:

1) Only search-time operations will be performed using the new sourcetype. No index-time settings associated with the new sourcetype will be applied.

2) Index-time settings associated with old sourcetype will still be applied!

[gcusello]

Hi @ITSplunk117 ,

yes it's possible to override the original sourcetype value with a new one using the procedure at https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Advancedsourcetypeoverrides

Only one attention pont: sourcetype overriding, as all transformations, must be performed on the first full Splunk instance that data re passing through, not necessarily on the Indexers.

In other words, if you have one or more intermediate heavy Forwarders, you must locate the transformation in the first Heavy Forwarder, not on the Indexers, because transformations are applied on the first Heavy Forwarder.

Ciao.

Giuseppe

[livehybrid · Accepted solution]

Hi @ITSplunk117 

This depends on if your data has already been through a heavy forwarder (HF) or only a Universal Forwarder.

If your data arrives at an indexer from a UF then you can indeed use the approach you've suggested (see updated example below)

== props.conf ==
[yourSourcetype]
TRANSFORMS-updatesourcetype = updateMySourcetype

== transforms.conf == 
[updateMySourcetype]
REGEX = .*
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::yourNewSourcetype

Regarding the FORMAT field, the word "sourcetype" is case-sensitive, the value after :: is less-so, the "DEST_KEY" needs to be as described above to update the sourcetype.

If the data comes from a heavy forwarder then you might still be able to achieve this with INGEST_EVAL

== props.conf ==
[yourSourcetype]
TRANSFORMS-updatesourcetype = updateMySourcetype

== transforms.conf == 
[updateMySourcetype]
INGEST_EVAL = sourcetype:=if(match(_raw,"SomeRegexHere"),"SourceTypeA","SourceTypeB")
= OR EVEN JUST ==
INGEST_EVAL = sourcetype:"NewSourcetypeName"

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[ITSplunk117]

Thank you that worked if the data arrives from a UF.  I had started with a HF which I could not change the sourcetype on, but that didn't surprise me either.  

We do not have access to the entire Splunk environment in every case and still waiting on a ticket to see if a custom sourcetype needs to be changed to a standard one.  

Most cases where there is a heavy forwarder it's likely we will have access to it and can take care of the sourcetype more directly through the endpoint.