Getting Data In

How to Split the below json file into multi events

Inayath_khan
Path Finder

Hi Folks,

Kindly help me to figure out dividing the below logs into each events.

     {
        "SecurityGroups": [
            {
                "Description": "default VPC security group",
                "GroupId": "abcd",
                "GroupName": "default",
                "IpPermissions": [
                    {
                        "FromPort": 23,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    },
                    {
                        "FromPort": 443,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0",
                                "Description": "For logs"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0",
                                "Description": "For logs"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [],
                "OwnerId": "##########",
                "VpcId": "vpc-#############"
            },
            {
                "Description": "launch-wizard-10 created 2019-03-27T18:10:47.749-07:00",
                "GroupId": "sg-#############",
                "GroupName": "launch-wizard-10",
                "IpPermissions": [
                    {
                        "FromPort": 22,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "VpcId": "vpc-01d67bc62cd3d2d62"
            },
            {
                "Description": "Security Group",
                "GroupId": "##############",
                "GroupName": "prd-#################",
                "IpPermissions": [
                    {
                        "FromPort": 8093,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/25",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 9093,
                        "UserIdGroupPairs": [
                            {
                                "Description": "########## ",
                                "GroupId": "##########",
                                "UserId": "#############"
                            }
                        ]
                    },
                    {
                        "FromPort": 9090,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/19",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 3899,
                        "UserIdGroupPairs": [
                            {
                                "Description": "######### ",
                                "GroupId": "############",
                                "UserId": "##############"
                            }
                        ]
                    },
                    {
                        "FromPort": 8080,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/23",
                                "Description": "#########"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": [
                            {
                                "Description": "HTTPS port",
                                "GroupId": "#############",
                                "UserId": "##############"
                            }
                        ]
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "Tags": [
                    {
                        "Key": "####",
                        "Value": "#######"
                    },
                    {
                        "Key": "#########",
                        "Value": "################"
                    },
                    {
                        "Key": "####",
                        "Value": "#####"
                    },
                    {
                        "Key": "Name",
                        "Value": "############"
                    }
                ],
                "VpcId": "vpc-#################"    
                }
           ]
   }

Awaiting for a quick response . Kindly reply on the post.

I tried the below method but failed at writing the REGEX.

https://answers.splunk.com/answers/642050/how-to-split-the-following-json-into-different-eve.html

Tags (2)
0 Karma
1 Solution

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

View solution in original post

0 Karma

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

0 Karma

martynoconnor
Communicator

How exactly do you want it split? What within the JSON posted do you consider an event breaker?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

What’s New in Splunk AI: Volume 02

Welcome to the second edition of “What’s New in Splunk AI” where we look at the latest and greatest updates, ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...