topic How to transform raw JSON with field called 'index' in Getting Data In

How to transform raw JSON with field called 'index'

jon_radford — Thu, 25 Nov 2021 11:48:04 GMT

Hi, I have json data being written to a log file and the log file is being forwarded to single Splunk index 'ti-l_asl'. The problem I have is the json data contains a field called 'index' which I want to transform into 'sourcetype' so it can be search on in Splunk. Is there a way I can do this without changing the system which writes the json to the log file i.e. transform the field name from 'index' to 'sourcetype' as part of the forwarder processing or some kind of pre-processing in Splunk before it is assigned to index 'ti-l_asl' ?

Re: How to transform raw JSON with field called 'index'

PickleRick — Thu, 25 Nov 2021 12:00:03 GMT

In general, if you leave the "index" field as is, you might have problems searching. Even if you add sourcetype based on that field value (which I suppose is possible using transforms and rewriting metadata), you'll still have a field called "index" in your event which will overlap with searching from a particular index.

Re: How to transform raw JSON with field called 'index'

jon_radford — Thu, 25 Nov 2021 13:08:13 GMT

Thanks @PickleRick

Having the index value in Splunk does not seem to affect searching, but I can only search on the Splunk index and not the data index value. How would I achieve the mapping using transforms or rewriting metadata?

Re: How to transform raw JSON with field called 'index'

PickleRick — Thu, 25 Nov 2021 13:33:25 GMT

That's what I'm talking about. You can't use the index field of the event to search. You just search by the index. 🙂

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Advancedsourcetypeoverrides

Re: How to transform raw JSON with field called 'index'

jon_radford — Thu, 25 Nov 2021 14:22:00 GMT

Thanks @PickleRick

So, to be clear, by using a heavy forwarder it could be used transform

{ "name" : "jon", "country" : "uk", "index" : "sit-event-2021-11-25" }

{"name" : "jon", "country" : "uk", "sourcetype" : "sit-event" }

using a regex to map any index value starting with "sit-event" to sourcetype "sit-event".

Alternatively this could be done on the instance if running Splunk Enterprise.

Have I understood this correctly?

Re: How to transform raw JSON with field called 'index'

PickleRick — Thu, 25 Nov 2021 14:29:15 GMT

You apply the transforms on the first "heavy" component that your data stream encounters. So if you have UF writing directly to indexer(s), you need to set the transform on indexer(s) but if you have more complicated environment like UF -> HF1 ->HF2 -> idx, you need to set it on first HF.

Oh, end you don't rewrite the data within the event itself, but in the event's metadata. Raw event stays as it was.

Re: How to transform raw JSON with field called 'index'

jon_radford — Thu, 25 Nov 2021 14:41:37 GMT

Thanks @PickleRick , will pass this solution onto our Splunk team to see if they can implement it as you described 👍