Data Model for hybrid type of log (like partial JSON or XML data)

[splunk_worker]

Hi

The portion of log is JSON. I shall extract the JSON portion using regex and pipe to 'spath input='. This will extract all the key-values from the JSON portion .

But, search takes lots of time due to extraction of 50+ of key-value pairs from JSON.

I have to write approx 10 search queries. So, instead of extracting the same in all 10 saved searches, it would be ideal to go for Data Model. If I extract all the fields in Data Model and accelerate in Root Event, then I shall write all 10 queries on these extracted fields on Child. Hence, all the 10 search queries will be faster.

My search query is
index=abc | rex "(?{[^}]+})" | mvexpand json_field | spath input=json_field

How to do put these commands in Data Model ?
1. I created root Event with "index=abc"
2. I have added a regular expression to extract JSON portion

I'm struck here. What is the next step? How to add spath command? Where to add?

I created a child with "spath input=json_field" and when try to create a pivot on Child object, I'm not seeing the key/fields of JSON portion. I mean, spath is don't success.

Please help me how to solve this problem.

Another question is, it will be great if we can apply 'spath' at root event level , so that, all the extracted fields are accelerated and all searches on child will be very faster.

Please advise.

[aneels_splunk · Accepted solution]

You can create a search-based root object with the search "index=abc | rex "(?<jsonfield>{[^}]+})" | mvexpand jsonfield | spath input=jsonfield", and have your other objects inherit from that. Unfortunately you currently can't use spath with an event-based object.

This has the limitation that it can't be accelerated (only event-based objects can be accelerated). But you will be able to use Data Model and the Pivot interface to encapsulate these searches and make them easier to manipulate. You can use the pivot search command to run searches against this data model, and save those searches if you wish.

[aneels_splunk]

If you don't need all of the fields from your JSON, you can use the spath command in a more targeted mode where you only extract one field at a time. Please refer to the spath docs: http://docs.splunk.com/Documentation/Splunk/6.1.1/SearchReference/Spath

[splunk_worker]

Thanks Aneels. 'spath' is time consuming command, it slows down the search result. Since we cant accelerate root-search, I'm not sure how to make the searches display the results faster.

[splunk_worker]

Is this possible in Data Model ?