Converting large JSON response into smaller events and combine objects

[david_rose]

I have a large JSON response that I need to split into smaller events as well as merge data from different points within the JSON into the single events. The JSON responses can be millions of lines, hence the need to split them into events.

Here is a small sample of a response. Real data has thousands of ip_address values:

{
    "policy_name": "1_External_SCAN_All_External_Subnets",
    "policy_type": "external",
    "fast_scan": false,
    "execution_date": 1413114210,
    "execution_time": 20220,
    "scan_result": "completed",
    "scan_summary": {
        "skipped": 0,
        "partial": 0,
        "scanned": 0,
        "rollover": 0
    },
    "errors": [],
    "total_hosts": 2144,
    "vulnerable_hosts": [
        {
            "ip_address": "127.0.0.1",
            "vulns": [
                {
                    "title": "TLS 1.0 & TLS 1.1 Weak Encryption Protocol",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "SSL - Server Supports Weak SSL Ciphers",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "CVE-2016-2183 - DES, Triple DES - Sweet32 Issue",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "SSL - Certificate Hostname Discrepancy",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "CVE-2013-2566 - RC4 - Plaintext-Recovery Issue",
                    "risk_level": "Medium",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "Web Service is Running",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 80
                },
                {
                    "title": "Web Service is Running",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "SSL Protocol - BEAST Attack - Server-Side Mitigation",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "3DES Detected",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 443
                },
                {
                    "title": "Web Service is Running",
                    "risk_level": "Low",
                    "service_protocol": "TCP",
                    "service_port": 8045
                }
            ]
        },
    ],
    "total_vulns": 7171,
    "vulns_summary": {
        "Critical": 0,
        "High": 296,
        "Low": 4345,
        "Medium": 2530,
        "Unclassified": 0,
        "Urgent": 0
    }
}

Each Vulns entry needs to be a seperate event. I also need the ip_address, policy_name, and execution date added to each event. I can split the events fine, but I have no idea how to attach the 3 extra fields to the events.

Sample event structure would look like:

"policy_name": "1_External_SCAN_All_External_Subnets"
"execution_date": 1413114210
"ip_address": "127.0.0.1"
"title": "TLS 1.0 & TLS 1.1 Weak Encryption Protocol",
"risk_level": "Medium",
"service_protocol": "TCP",
"service_port": 443