Solved: Extract JSON data within the logs ( JSON mixed wit... - Page 2

koshyk · ‎01-03-2014

We got a requirement to extract information from log file. The log file contains JSON data which is the bread-butter for splunk. This is a mixed data whereby the logging application puts some info like logging time| messageSeverity | class | thread etc..

Later, the JSON message starts like - [{ json }].

2013-12-23T14:55:09.574+0000|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=102;_ThreadName=Thread-2;|2013-12-23 14:55:09,574 DEBUG parent-container$child#1-10 [] com.abc.transform.listeners.xyz- [{
  "timestamp" : "2013-12-23T14:55:09.558Z",
  "host" : "myPC",
  "event_id" : "1234",
  "customer_id" : "123456",
...
...

  "country" : "Canada",
  "product" : "iPad",
  "msg" : "Hello Guys",
  "transaction_id" : "100200300400"
  }
}]
|

Please note that this JSON is not fixed, so it can extend to extra lines.
How to extract the JSON data alone into key-value pairs for easy presentation?

koshyk · ‎12-03-2016

The high level idea to do it automatically within props.conf and transforms.conf is something like below..

To extract JSON keyvalue from mixed data

The 1st Transform will extract "json1" and then subsequently do more transforms

### inputs.conf
[monitor:///var/log/json.log]
sourcetype = myjson

### props.conf
[myjson]
REPORT-json = report-json,report-json-kv

transforms.conf

[report-json]       
# This will get the json payload from the logs. 
# Put your specific logic if you need. Below is a very basic logic baed on { bracket
REGEX = (?P<json1>{.+)

# Manually extract JSON key-value
[report-json-kv]       
REGEX = \"(\w+)\":[\s]*\"([^\,\}\"]+)
FORMAT = $1::$2
MV_ADD = true

View solution in original post

Extract JSON data within the logs ( JSON mixed with unstructured data)

To extract JSON keyvalue from mixed data

transforms.conf

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Supercharging Windows Security Detection Performance: Introducing Hybrid Field ...

Ditch the Manual Grind: Building AI Agents with Splunk

Cisco Data Fabric from Architecture to Investigation, Better SOC Visibility, and More ...

Join the Conversation

Extract JSON data within the logs ( JSON mixed with unstructured data)

To extract JSON keyvalue from mixed data

transforms.conf

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Supercharging Windows Security Detection Performance: Introducing Hybrid Field ...

Ditch the Manual Grind: Building AI Agents with Splunk

Cisco Data Fabric from Architecture to Investigation, Better SOC Visibility, and More ...