Getting Data In

How to Split the below json file into multi events

Inayath_khan
Path Finder

Hi Folks,

Kindly help me to figure out dividing the below logs into each events.

     {
        "SecurityGroups": [
            {
                "Description": "default VPC security group",
                "GroupId": "abcd",
                "GroupName": "default",
                "IpPermissions": [
                    {
                        "FromPort": 23,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    },
                    {
                        "FromPort": 443,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0",
                                "Description": "For logs"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0",
                                "Description": "For logs"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [],
                "OwnerId": "##########",
                "VpcId": "vpc-#############"
            },
            {
                "Description": "launch-wizard-10 created 2019-03-27T18:10:47.749-07:00",
                "GroupId": "sg-#############",
                "GroupName": "launch-wizard-10",
                "IpPermissions": [
                    {
                        "FromPort": 22,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "VpcId": "vpc-01d67bc62cd3d2d62"
            },
            {
                "Description": "Security Group",
                "GroupId": "##############",
                "GroupName": "prd-#################",
                "IpPermissions": [
                    {
                        "FromPort": 8093,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/25",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 9093,
                        "UserIdGroupPairs": [
                            {
                                "Description": "########## ",
                                "GroupId": "##########",
                                "UserId": "#############"
                            }
                        ]
                    },
                    {
                        "FromPort": 9090,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/19",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 3899,
                        "UserIdGroupPairs": [
                            {
                                "Description": "######### ",
                                "GroupId": "############",
                                "UserId": "##############"
                            }
                        ]
                    },
                    {
                        "FromPort": 8080,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/23",
                                "Description": "#########"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": [
                            {
                                "Description": "HTTPS port",
                                "GroupId": "#############",
                                "UserId": "##############"
                            }
                        ]
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "Tags": [
                    {
                        "Key": "####",
                        "Value": "#######"
                    },
                    {
                        "Key": "#########",
                        "Value": "################"
                    },
                    {
                        "Key": "####",
                        "Value": "#####"
                    },
                    {
                        "Key": "Name",
                        "Value": "############"
                    }
                ],
                "VpcId": "vpc-#################"    
                }
           ]
   }

Awaiting for a quick response . Kindly reply on the post.

I tried the below method but failed at writing the REGEX.

https://answers.splunk.com/answers/642050/how-to-split-the-following-json-into-different-eve.html

Tags (2)
0 Karma
1 Solution

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

View solution in original post

0 Karma

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

0 Karma

martynoconnor
Communicator

How exactly do you want it split? What within the JSON posted do you consider an event breaker?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...