Solved: Extract JSON data within the logs ( JSON mixed wit... - Page 2

koshyk · ‎01-03-2014

We got a requirement to extract information from log file. The log file contains JSON data which is the bread-butter for splunk. This is a mixed data whereby the logging application puts some info like logging time| messageSeverity | class | thread etc..

Later, the JSON message starts like - [{ json }].

2013-12-23T14:55:09.574+0000|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=102;_ThreadName=Thread-2;|2013-12-23 14:55:09,574 DEBUG parent-container$child#1-10 [] com.abc.transform.listeners.xyz- [{
  "timestamp" : "2013-12-23T14:55:09.558Z",
  "host" : "myPC",
  "event_id" : "1234",
  "customer_id" : "123456",
...
...

  "country" : "Canada",
  "product" : "iPad",
  "msg" : "Hello Guys",
  "transaction_id" : "100200300400"
  }
}]
|

Please note that this JSON is not fixed, so it can extend to extra lines.
How to extract the JSON data alone into key-value pairs for easy presentation?

koshyk · ‎12-03-2016

The high level idea to do it automatically within props.conf and transforms.conf is something like below..

To extract JSON keyvalue from mixed data

The 1st Transform will extract "json1" and then subsequently do more transforms

### inputs.conf
[monitor:///var/log/json.log]
sourcetype = myjson

### props.conf
[myjson]
REPORT-json = report-json,report-json-kv

transforms.conf

[report-json]       
# This will get the json payload from the logs. 
# Put your specific logic if you need. Below is a very basic logic baed on { bracket
REGEX = (?P<json1>{.+)

# Manually extract JSON key-value
[report-json-kv]       
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
MV_ADD = true

View solution in original post

Extract JSON data within the logs ( JSON mixed with unstructured data)

To extract JSON keyvalue from mixed data

transforms.conf

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Defend at Machine Speed: Your Guide to Security Sessions at .conf26

Where Innovation Takes Flight: The Splunk4Aviation Flight Sim Lands at .conf26

Turn Cisco Telemetry Into Action with Cisco Data Fabric, powered by the Splunk ...

Join the Conversation

Extract JSON data within the logs ( JSON mixed with unstructured data)

To extract JSON keyvalue from mixed data

transforms.conf

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Defend at Machine Speed: Your Guide to Security Sessions at .conf26

Where Innovation Takes Flight: The Splunk4Aviation Flight Sim Lands at .conf26

Turn Cisco Telemetry Into Action with Cisco Data Fabric, powered by the Splunk ...