Getting Data In

How to Split the below json file into multi events

Inayath_khan
Path Finder

Hi Folks,

Kindly help me to figure out dividing the below logs into each events.

     {
        "SecurityGroups": [
            {
                "Description": "default VPC security group",
                "GroupId": "abcd",
                "GroupName": "default",
                "IpPermissions": [
                    {
                        "FromPort": 23,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    },
                    {
                        "FromPort": 443,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0",
                                "Description": "For logs"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0",
                                "Description": "For logs"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [],
                "OwnerId": "##########",
                "VpcId": "vpc-#############"
            },
            {
                "Description": "launch-wizard-10 created 2019-03-27T18:10:47.749-07:00",
                "GroupId": "sg-#############",
                "GroupName": "launch-wizard-10",
                "IpPermissions": [
                    {
                        "FromPort": 22,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "VpcId": "vpc-01d67bc62cd3d2d62"
            },
            {
                "Description": "Security Group",
                "GroupId": "##############",
                "GroupName": "prd-#################",
                "IpPermissions": [
                    {
                        "FromPort": 8093,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/25",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 9093,
                        "UserIdGroupPairs": [
                            {
                                "Description": "########## ",
                                "GroupId": "##########",
                                "UserId": "#############"
                            }
                        ]
                    },
                    {
                        "FromPort": 9090,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/19",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 3899,
                        "UserIdGroupPairs": [
                            {
                                "Description": "######### ",
                                "GroupId": "############",
                                "UserId": "##############"
                            }
                        ]
                    },
                    {
                        "FromPort": 8080,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/23",
                                "Description": "#########"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": [
                            {
                                "Description": "HTTPS port",
                                "GroupId": "#############",
                                "UserId": "##############"
                            }
                        ]
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "Tags": [
                    {
                        "Key": "####",
                        "Value": "#######"
                    },
                    {
                        "Key": "#########",
                        "Value": "################"
                    },
                    {
                        "Key": "####",
                        "Value": "#####"
                    },
                    {
                        "Key": "Name",
                        "Value": "############"
                    }
                ],
                "VpcId": "vpc-#################"    
                }
           ]
   }

Awaiting for a quick response . Kindly reply on the post.

I tried the below method but failed at writing the REGEX.

https://answers.splunk.com/answers/642050/how-to-split-the-following-json-into-different-eve.html

Tags (2)
0 Karma
1 Solution

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

View solution in original post

0 Karma

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

0 Karma

martynoconnor
Communicator

How exactly do you want it split? What within the JSON posted do you consider an event breaker?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...