Getting Data In

How to break a json log into multiple events

mah
Builder

Hi,

I have difficulty to break a json into multiple events.

Here is my log : (appear in one event, instead of 2)

 

{
    "InstanceInformationList": [
        {
            "Version": false, 
            "PlatformName": "Amazon Linux", 
            "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",  
            "IPAddress": "10.170.216.18", 
            "AssociationOverview": {
                "DetailedStatus": "Failed", 
                "InstanceAssociationStatusAggregatedCount": {
                    "Failed": 1, 
                    "Success": 1
                }
            }, 
            "AssociationStatus": "Failed",  
            "PlatformVersion": "2", 
            "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", 
            "InstanceId": "i-00000000001", 
            "PlatformName": "Amazon Linux"
        }
    ]
}

 

 

 And you can find my props.conf below :

 

[my_test]
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = json
DATETIME_CONFIG = CURRENT
TRUNCATE = 999999
JSON_TRIM_BRACES_IN_ARRAY_NAMES = true
BREAK_ONLY_BEFORE = (\[\s+\{)
MUST_BREAK_AFTER = (\},|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_footer = s/\]\s+\}//g

 

 

 

Can you help me to find the write parsing please ?

Thank you.

Labels (2)
0 Karma

mah
Builder

Hi @richgalloway 

I want to break where I have : closed braces / comma / opened braces :

   "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",  

Thanks a lot for your help !

 

 

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

You could try this.

| makeresults 
| eval _raw = "{
    \"InstanceInformationList\": [
        {
            \"Version\": false,
            \"PlatformName\": \"Amazon Linux\",
            \"ComputerName\": \"ip-10-170-216-17.eu-east-1.compute.internal\"
        },
        {
            \"PlatformType\": \"Linux\",
            \"IPAddress\": \"10.170.216.18\",
            \"AssociationOverview\": {
                \"DetailedStatus\": \"Failed\",
                \"InstanceAssociationStatusAggregatedCount\": {
                    \"Failed\": 1,
                    \"Success\": 1
                }
            },
            \"AssociationStatus\": \"Failed\",
            \"PlatformVersion\": \"2\",
            \"ComputerName\": \"ip-10-170-216-18.eu-east-1.compute.internal\",
            \"InstanceId\": \"i-00000000001\",
            \"PlatformName\": \"Amazon Linux\"
        }
    ]
}"
| spath output=Computer path=InstanceInformationList{}.ComputerName

Using above you could get mv field Computer where are those ComputerNames.

r. Ismo

0 Karma

mah
Builder

I want to break the json at indextime because mvexpand command is ressource consuming. 

I want  to separate the json in order to have 1 braces by event : 

One event :

 { "Version": false, "PlatformName": "Amazon Linux", "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal" }

another event :

{ "PlatformType": "Linux", "IPAddress": "10.170.216.18", "AssociationOverview": { "DetailedStatus": "Failed", "InstanceAssociationStatusAggregatedCount": { "Failed": 1, "Success": 1 } }, "AssociationStatus": "Failed", "PlatformVersion": "2", "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", "InstanceId": "i-00000000001", "PlatformName": "Amazon Linux" }

Wherever there is this : }, {  split the json by using setting in props.conf

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

Are you really sure that you want break that JSON to several events on indexing time? If you do it then you cannot manage it as whole JSON instance later on?

If this is really what you want to do, you could try this

DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
LINE_BREAKER=(\s+\[\s+\{)|(\s+\},\s+\{\s+)|(\}\s+\])
NO_BINARY_CHECK=true
TRUNCATE=0

Basically you must forget that this is JSON and manage it as normal text event.

I think that you must use separate transforms.conf to get rid of those "unnecessary" header and footer. Based on order which props + transforms are handled you cannot use SEDCMD in this case. 

r. Ismo

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where do you want to break the event?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...