topic Re: How to break a json log into multiple events in Getting Data In

How to break a json log into multiple events

mah — Wed, 06 Oct 2021 15:22:54 GMT

Hi,

I have difficulty to break a json into multiple events.

Here is my log : (appear in one event, instead of 2)

{ "InstanceInformationList": [ { "Version": false, "PlatformName": "Amazon Linux", "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal" }, { "PlatformType": "Linux", "IPAddress": "10.170.216.18", "AssociationOverview": { "DetailedStatus": "Failed", "InstanceAssociationStatusAggregatedCount": { "Failed": 1, "Success": 1 } }, "AssociationStatus": "Failed", "PlatformVersion": "2", "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", "InstanceId": "i-00000000001", "PlatformName": "Amazon Linux" } ] }

And you can find my props.conf below :

[my_test] SHOULD_LINEMERGE = false INDEXED_EXTRACTIONS = json DATETIME_CONFIG = CURRENT TRUNCATE = 999999 JSON_TRIM_BRACES_IN_ARRAY_NAMES = true BREAK_ONLY_BEFORE = (\[\s+\{) MUST_BREAK_AFTER = (\},|\}\s+\]) SEDCMD-remove_header = s/(\{\s+.+?\[)//g SEDCMD-remove_footer = s/\]\s+\}//g

Can you help me to find the write parsing please ?

Thank you.

Re: How to break a json log into multiple events

richgalloway — Wed, 06 Oct 2021 17:22:44 GMT

Where do you want to break the event?

Re: How to break a json log into multiple events

mah — Thu, 07 Oct 2021 07:21:38 GMT

Hi @richgalloway

I want to break where I have : closed braces / comma / opened braces :

   "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",

Thanks a lot for your help !

Re: How to break a json log into multiple events

isoutamo — Thu, 07 Oct 2021 07:45:47 GMT

You could try this.

| makeresults | eval _raw = "{ \"InstanceInformationList\": [ { \"Version\": false, \"PlatformName\": \"Amazon Linux\", \"ComputerName\": \"ip-10-170-216-17.eu-east-1.compute.internal\" }, { \"PlatformType\": \"Linux\", \"IPAddress\": \"10.170.216.18\", \"AssociationOverview\": { \"DetailedStatus\": \"Failed\", \"InstanceAssociationStatusAggregatedCount\": { \"Failed\": 1, \"Success\": 1 } }, \"AssociationStatus\": \"Failed\", \"PlatformVersion\": \"2\", \"ComputerName\": \"ip-10-170-216-18.eu-east-1.compute.internal\", \"InstanceId\": \"i-00000000001\", \"PlatformName\": \"Amazon Linux\" } ] }" | spath output=Computer path=InstanceInformationList{}.ComputerName

Using above you could get mv field Computer where are those ComputerNames.

r. Ismo

Re: How to break a json log into multiple events

mah — Thu, 07 Oct 2021 07:59:31 GMT

I want to break the json at indextime because mvexpand command is ressource consuming.

I want to separate the json in order to have 1 braces by event :

One event :

{ "Version": false, "PlatformName": "Amazon Linux", "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal" }

another event :

{ "PlatformType": "Linux", "IPAddress": "10.170.216.18", "AssociationOverview": { "DetailedStatus": "Failed", "InstanceAssociationStatusAggregatedCount": { "Failed": 1, "Success": 1 } }, "AssociationStatus": "Failed", "PlatformVersion": "2", "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", "InstanceId": "i-00000000001", "PlatformName": "Amazon Linux" }

Wherever there is this : }, { split the json by using setting in props.conf

Re: How to break a json log into multiple events

isoutamo — Wed, 13 Oct 2021 09:00:30 GMT

Are you really sure that you want break that JSON to several events on indexing time? If you do it then you cannot manage it as whole JSON instance later on?

If this is really what you want to do, you could try this

DATETIME_CONFIG=CURRENT SHOULD_LINEMERGE=true LINE_BREAKER=(\s+\[\s+\{)|(\s+\},\s+\{\s+)|(\}\s+\]) NO_BINARY_CHECK=true TRUNCATE=0

Basically you must forget that this is JSON and manage it as normal text event.

I think that you must use separate transforms.conf to get rid of those "unnecessary" header and footer. Based on order which props + transforms are handled you cannot use SEDCMD in this case.

r. Ismo