Getting Data In

How to Split the below json file into multi events

Inayath_khan
Path Finder

Hi Folks,

Kindly help me to figure out dividing the below logs into each events.

     {
        "SecurityGroups": [
            {
                "Description": "default VPC security group",
                "GroupId": "abcd",
                "GroupName": "default",
                "IpPermissions": [
                    {
                        "FromPort": 23,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    },
                    {
                        "FromPort": 443,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0",
                                "Description": "For logs"
                            }
                        ],
                        "Ipv6Ranges": [
                            {
                                "CidrIpv6": "::/0",
                                "Description": "For logs"
                            }
                        ],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [],
                "OwnerId": "##########",
                "VpcId": "vpc-#############"
            },
            {
                "Description": "launch-wizard-10 created 2019-03-27T18:10:47.749-07:00",
                "GroupId": "sg-#############",
                "GroupName": "launch-wizard-10",
                "IpPermissions": [
                    {
                        "FromPort": 22,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 22,
                        "UserIdGroupPairs": []
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "VpcId": "vpc-01d67bc62cd3d2d62"
            },
            {
                "Description": "Security Group",
                "GroupId": "##############",
                "GroupName": "prd-#################",
                "IpPermissions": [
                    {
                        "FromPort": 8093,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/25",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 9093,
                        "UserIdGroupPairs": [
                            {
                                "Description": "########## ",
                                "GroupId": "##########",
                                "UserId": "#############"
                            }
                        ]
                    },
                    {
                        "FromPort": 9090,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/19",
                                "Description": "############"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 3899,
                        "UserIdGroupPairs": [
                            {
                                "Description": "######### ",
                                "GroupId": "############",
                                "UserId": "##############"
                            }
                        ]
                    },
                    {
                        "FromPort": 8080,
                        "IpProtocol": "tcp",
                        "IpRanges": [
                            {
                                "CidrIp": "10.0.0.1/23",
                                "Description": "#########"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "ToPort": 443,
                        "UserIdGroupPairs": [
                            {
                                "Description": "HTTPS port",
                                "GroupId": "#############",
                                "UserId": "##############"
                            }
                        ]
                    }
                ],
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "OwnerId": "439086027177",
                "Tags": [
                    {
                        "Key": "####",
                        "Value": "#######"
                    },
                    {
                        "Key": "#########",
                        "Value": "################"
                    },
                    {
                        "Key": "####",
                        "Value": "#####"
                    },
                    {
                        "Key": "Name",
                        "Value": "############"
                    }
                ],
                "VpcId": "vpc-#################"    
                }
           ]
   }

Awaiting for a quick response . Kindly reply on the post.

I tried the below method but failed at writing the REGEX.

https://answers.splunk.com/answers/642050/how-to-split-the-following-json-into-different-eve.html

Tags (2)
0 Karma
1 Solution

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

View solution in original post

0 Karma

martynoconnor
Communicator

The lack of a timestamp is an issue - cause if you're reading all of these from a single JSON, you'll have Splunk thinking everything happened at once, at the point of ingest.

Anyway, you can break the JSON into events using BREAK_ONLY_BEFORE = \"Description\":\s+\"[\w\s]+\" in your sourcetype definition and by disabling INDEXED_EXTRACTIONS = json, but then by telling Splunk to ignore its normal parsing of JSON, you lose all of the extracted fields, and you'll need to use EXTRACT-blah in props to pull things out.

0 Karma

martynoconnor
Communicator

How exactly do you want it split? What within the JSON posted do you consider an event breaker?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...