How to transform raw JSON with field called 'index'

[jon_radford]

Hi, I have json data being written to a log file and the log file is being forwarded to single Splunk index 'ti-l_asl'. The problem I have is the json data contains a field called 'index' which I want to transform into 'sourcetype' so it can be search on in Splunk. Is there a way I can do this without changing the system which writes the json to the log file i.e. transform the field name from 'index' to 'sourcetype' as part of the forwarder processing or some kind of pre-processing in Splunk before it is assigned to index 'ti-l_asl' ?

[PickleRick]

In general, if you leave the "index" field as is, you might have problems searching. Even if you add sourcetype based on that field value (which I suppose is possible using transforms and rewriting metadata), you'll still have a field called "index" in your event which will overlap with searching from a particular index.

[jon_radford]

Thanks @PickleRick 

Having the index value in Splunk does not seem to affect searching, but I can only search on the Splunk index and not the data index value. How would I achieve the mapping using transforms or rewriting metadata?

[PickleRick]

That's what I'm talking about. You can't use the index field of the event to search. You just search by the index. 🙂

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Advancedsourcetypeoverrides

[jon_radford]

Thanks @PickleRick 

So, to be clear, by using a heavy forwarder it could be used transform

{ "name" : "jon", "country" : "uk", "index" : "sit-event-2021-11-25" }

to

{"name" : "jon", "country" : "uk", "sourcetype" : "sit-event" }

using a regex to map any index value starting with "sit-event" to sourcetype "sit-event".

Alternatively this could be done on the instance if running Splunk Enterprise.

Have I understood this correctly?

[PickleRick]

You apply the transforms on the first "heavy" component that your data stream encounters. So if you have UF writing directly to indexer(s), you need to set the transform on indexer(s) but if you have more complicated environment like UF -> HF1 ->HF2 -> idx, you need to set it on first HF.

Oh, end you don't rewrite the data within the event itself, but in the event's metadata. Raw event stays as it was.

[jon_radford]

Thanks @PickleRick , will pass this solution onto our Splunk team to see if they can implement it as you described 👍