Getting Data In
Highlighted

Extract JSON data within the logs ( JSON mixed with unstructured data)

Super Champion

We got a requirement to extract information from log file. The log file contains JSON data which is the bread-butter for splunk. This is a mixed data whereby the logging application puts some info like logging time| messageSeverity | class | thread etc..

Later, the JSON message starts like - [{ json }].

2013-12-23T14:55:09.574+0000|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=102;_ThreadName=Thread-2;|2013-12-23 14:55:09,574 DEBUG parent-container$child#1-10 [] com.abc.transform.listeners.xyz- [{
  "timestamp" : "2013-12-23T14:55:09.558Z",
  "host" : "myPC",
  "event_id" : "1234",
  "customer_id" : "123456",
...
...

  "country" : "Canada",
  "product" : "iPad",
  "msg" : "Hello Guys",
  "transaction_id" : "100200300400"
  }
}]
|

Please note that this JSON is not fixed, so it can extend to extra lines.
How to extract the JSON data alone into key-value pairs for easy presentation?

Tags (2)
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

SplunkTrust
SplunkTrust
0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

Super Champion

I agree if its pure JSON data. But the above entry is a mix of traditional log info + JSON

0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

SplunkTrust
SplunkTrust

You can extract the JSON part into a field and then run spath from that as an input field:

spath [input=<field>] [output=<field>] [path=<datapath> | <datapath>]
0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

Explorer

@koshyk Where you ever able to solve this? I would be interested in how if so.

0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

SplunkTrust
SplunkTrust

From your event, extract the JSON part to a field and then do spath to process that. For example from your event extracted a filed my_data using rex and then pass it to spath

     2013-12-23T14:55:09.574+0000|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=102;_ThreadName=Thread-2;|2013-12-23 14:55:09,574 DEBUG parent-container$child#1-10 [] com.abc.transform.listeners.xyz- [{   "timestamp" : "2013-12-23T14:55:09.558Z",   "host" : "myPC",   "event_id" : "1234",   "customer_id" : "123456",   "country" : "Canada",   "product" : "iPad",   "msg" : "Hello Guys",   "transaction_id" : "100200300400"   }]

Step by step

index=* sourcetype=json_data|rex "^(?:[^ \n]* ){7}(?P<my_data>.+)"|table my_data

You should be able to see only your JSON string there, if not adjust the regex according to your requirement.

Then parse it with spath

index=* sourcetype=json_data|rex "^(?:[^ \n]* ){7}(?P<my_data>.+)"|table my_data|spath input=my_data

You should be able to see all your fields there.

Now rename/reuse it for further processing. For example

index=* sourcetype=json_data|rex "^(?:[^ \n]* ){7}(?P<my_data>.+)"|spath input=my_data|rename {}.host as MY_HOST,{}.event_id as MY_EVENT|table MY_HOST MY_EVENT

Hope this helps

Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

Path Finder

How can this be dont automatically? eval with spath in props.conf only breaks out specific field in the spath path. I'm dealing with hybrid logs like this too.

0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

Super Champion

hi mate, inorder to do this automatically, you need to have "props.conf" and "transforms.conf" and the put the above logic. There are lot of examples in splunk.answers. If not , let me know and I can create an example.cheers

0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

Explorer

I have not found any examples of how to extract nested json automatically in prop.conf/transforms.conf. If you have such examples it would be much appreciated.

0 Karma
Highlighted

Re: Extract JSON data within the logs ( JSON mixed with unstructured data)

Super Champion

hi mate, the accepted answer above will do the exact same thing.
report-json => This will extract pure json message from the mixed message. It should be your logic
report-json-kv => This will extract json (nested) from pure json message

0 Karma