Getting Data In

How to transform raw JSON with field called 'index'

jon_radford
Explorer
Hi, I have json data being written to a log file and the log file is being forwarded to single Splunk index 'ti-l_asl'. The problem I have is the json data contains a field called 'index' which I want to transform into 'sourcetype' so it can be search on in Splunk. Is there a way I can do this without changing the system which writes the json to the log file i.e. transform the field name from 'index' to 'sourcetype' as part of the forwarder processing or some kind of pre-processing in Splunk before it is assigned to index 'ti-l_asl' ?
Tags (1)
0 Karma

PickleRick
Champion

In general, if you leave the "index" field as is, you might have problems searching. Even if you add sourcetype based on that field value (which I suppose is possible using transforms and rewriting metadata), you'll still have a field called "index" in your event which will overlap with searching from a particular index.

 

jon_radford
Explorer

Thanks @PickleRick 

Having the index value in Splunk does not seem to affect searching, but I can only search on the Splunk index and not the data index value. How would I achieve the mapping using transforms or rewriting metadata?

0 Karma

PickleRick
Champion

That's what I'm talking about. You can't use the index field of the event to search. You just search by the index. 🙂

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Advancedsourcetypeoverrides

jon_radford
Explorer

Thanks @PickleRick 

So, to be clear, by using a heavy forwarder it could be used transform

{ "name" : "jon", "country" : "uk", "index" : "sit-event-2021-11-25" }

to

{"name" : "jon", "country" : "uk", "sourcetype" : "sit-event" }

using a regex to map any index value starting with "sit-event" to sourcetype "sit-event".

Alternatively this could be done on the instance if running Splunk Enterprise.

Have I understood this correctly?

0 Karma

PickleRick
Champion

You apply the transforms on the first "heavy" component that your data stream encounters. So if you have UF writing directly to indexer(s), you need to set the transform on indexer(s) but if you have more complicated environment like UF -> HF1 ->HF2 -> idx, you need to set it on first HF.

Oh, end you don't rewrite the data within the event itself, but in the event's metadata. Raw event stays as it was.

jon_radford
Explorer

Thanks @PickleRick , will pass this solution onto our Splunk team to see if they can implement it as you described 👍

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!