Getting Data In

How to transform raw JSON with field called 'index'

jon_radford
Explorer
Hi, I have json data being written to a log file and the log file is being forwarded to single Splunk index 'ti-l_asl'. The problem I have is the json data contains a field called 'index' which I want to transform into 'sourcetype' so it can be search on in Splunk. Is there a way I can do this without changing the system which writes the json to the log file i.e. transform the field name from 'index' to 'sourcetype' as part of the forwarder processing or some kind of pre-processing in Splunk before it is assigned to index 'ti-l_asl' ?
Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

In general, if you leave the "index" field as is, you might have problems searching. Even if you add sourcetype based on that field value (which I suppose is possible using transforms and rewriting metadata), you'll still have a field called "index" in your event which will overlap with searching from a particular index.

 

jon_radford
Explorer

Thanks @PickleRick 

Having the index value in Splunk does not seem to affect searching, but I can only search on the Splunk index and not the data index value. How would I achieve the mapping using transforms or rewriting metadata?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

That's what I'm talking about. You can't use the index field of the event to search. You just search by the index. 🙂

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Advancedsourcetypeoverrides

jon_radford
Explorer

Thanks @PickleRick 

So, to be clear, by using a heavy forwarder it could be used transform

{ "name" : "jon", "country" : "uk", "index" : "sit-event-2021-11-25" }

to

{"name" : "jon", "country" : "uk", "sourcetype" : "sit-event" }

using a regex to map any index value starting with "sit-event" to sourcetype "sit-event".

Alternatively this could be done on the instance if running Splunk Enterprise.

Have I understood this correctly?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You apply the transforms on the first "heavy" component that your data stream encounters. So if you have UF writing directly to indexer(s), you need to set the transform on indexer(s) but if you have more complicated environment like UF -> HF1 ->HF2 -> idx, you need to set it on first HF.

Oh, end you don't rewrite the data within the event itself, but in the event's metadata. Raw event stays as it was.

jon_radford
Explorer

Thanks @PickleRick , will pass this solution onto our Splunk team to see if they can implement it as you described 👍

0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...