How to break a json log into multiple events

[mah]

Hi,

I have difficulty to break a json into multiple events.

Here is my log : (appear in one event, instead of 2)

{
    "InstanceInformationList": [
        {
            "Version": false, 
            "PlatformName": "Amazon Linux", 
            "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",  
            "IPAddress": "10.170.216.18", 
            "AssociationOverview": {
                "DetailedStatus": "Failed", 
                "InstanceAssociationStatusAggregatedCount": {
                    "Failed": 1, 
                    "Success": 1
                }
            }, 
            "AssociationStatus": "Failed",  
            "PlatformVersion": "2", 
            "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", 
            "InstanceId": "i-00000000001", 
            "PlatformName": "Amazon Linux"
        }
    ]
}

 And you can find my props.conf below :

[my_test]
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = json
DATETIME_CONFIG = CURRENT
TRUNCATE = 999999
JSON_TRIM_BRACES_IN_ARRAY_NAMES = true
BREAK_ONLY_BEFORE = (\[\s+\{)
MUST_BREAK_AFTER = (\},|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_footer = s/\]\s+\}//g

Can you help me to find the write parsing please ?

Thank you.

[mah]

Hi @richgalloway 

I want to break where I have : closed braces / comma / opened braces :

   "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",  

Thanks a lot for your help !

[isoutamo]

Hi

You could try this.

| makeresults 
| eval _raw = "{
    \"InstanceInformationList\": [
        {
            \"Version\": false,
            \"PlatformName\": \"Amazon Linux\",
            \"ComputerName\": \"ip-10-170-216-17.eu-east-1.compute.internal\"
        },
        {
            \"PlatformType\": \"Linux\",
            \"IPAddress\": \"10.170.216.18\",
            \"AssociationOverview\": {
                \"DetailedStatus\": \"Failed\",
                \"InstanceAssociationStatusAggregatedCount\": {
                    \"Failed\": 1,
                    \"Success\": 1
                }
            },
            \"AssociationStatus\": \"Failed\",
            \"PlatformVersion\": \"2\",
            \"ComputerName\": \"ip-10-170-216-18.eu-east-1.compute.internal\",
            \"InstanceId\": \"i-00000000001\",
            \"PlatformName\": \"Amazon Linux\"
        }
    ]
}"
| spath output=Computer path=InstanceInformationList{}.ComputerName

Using above you could get mv field Computer where are those ComputerNames.

r. Ismo

[mah]

I want to break the json at indextime because mvexpand command is ressource consuming. 

I want  to separate the json in order to have 1 braces by event : 

One event :

 { "Version": false, "PlatformName": "Amazon Linux", "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal" }

another event :

{ "PlatformType": "Linux", "IPAddress": "10.170.216.18", "AssociationOverview": { "DetailedStatus": "Failed", "InstanceAssociationStatusAggregatedCount": { "Failed": 1, "Success": 1 } }, "AssociationStatus": "Failed", "PlatformVersion": "2", "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", "InstanceId": "i-00000000001", "PlatformName": "Amazon Linux" }

Wherever there is this : }, {  split the json by using setting in props.conf

[isoutamo]

Hi

Are you really sure that you want break that JSON to several events on indexing time? If you do it then you cannot manage it as whole JSON instance later on?

If this is really what you want to do, you could try this

DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
LINE_BREAKER=(\s+\[\s+\{)|(\s+\},\s+\{\s+)|(\}\s+\])
NO_BINARY_CHECK=true
TRUNCATE=0

Basically you must forget that this is JSON and manage it as normal text event.

I think that you must use separate transforms.conf to get rid of those "unnecessary" header and footer. Based on order which props + transforms are handled you cannot use SEDCMD in this case. 

r. Ismo

[richgalloway]

Where do you want to break the event?