Getting Data In

Converting large JSON response into smaller events and combine objects

david_rose
Communicator

I have a large JSON response that I need to split into smaller events as well as merge data from different points within the JSON into the single events. The JSON responses can be millions of lines, hence the need to split them into events.

Here is a small sample of a response. Real data has thousands of ip_address values:

{
    "policy_name": "1_External_SCAN_All_External_Subnets",
    "policy_type": "external",
    "fast_scan": false,
    "execution_date": 1413114210,
    "execution_time": 20220,
    "scan_result": "completed",
    "scan_summary": {
        "skipped": 0,
        "partial": 0,
        "scanned": 0,
        "rollover": 0
    },
    "errors": [],
    "total_hosts": 2144,
    "vulnerable_hosts": [
        {
            "ip_address": "127.0.0.1",
            "vulns": [
                {
                    "title": "TLS 1.0 & TLS 1.1 Weak Encryption Protocol",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "SSL - Server Supports Weak SSL Ciphers",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "CVE-2016-2183 - DES, Triple DES - Sweet32 Issue",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "SSL - Certificate Hostname Discrepancy",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "CVE-2013-2566 - RC4 - Plaintext-Recovery Issue",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "Web Service is Running",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 80
                },
                {
                    "title": "Web Service is Running",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "SSL Protocol - BEAST Attack - Server-Side Mitigation",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "3DES Detected",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "Web Service is Running",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 8045
                }
            ]
        },
    ],
    "total_vulns": 7171,
    "vulns_summary": {
        "Critical": 0,
        "High": 296,
        "Low": 4345,
        "Medium": 2530,
        "Unclassified": 0,
        "Urgent": 0
    }
}

Each Vulns entry needs to be a seperate event. I also need the ip_address, policy_name, and execution date added to each event. I can split the events fine, but I have no idea how to attach the 3 extra fields to the events.

Sample event structure would look like:

"policy_name": "1_External_SCAN_All_External_Subnets"
"execution_date": 1413114210
"ip_address": "127.0.0.1"
"title": "TLS 1.0 & TLS 1.1 Weak Encryption Protocol",
"risk_level": "Medium",
"service_protocol": "TCP",
"service_port": 443
Tags (2)
0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...