Getting Data In

How to break a json log into multiple events

mah
Builder

Hi,

I have difficulty to break a json into multiple events.

Here is my log : (appear in one event, instead of 2)

 

{
    "InstanceInformationList": [
        {
            "Version": false, 
            "PlatformName": "Amazon Linux", 
            "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",  
            "IPAddress": "10.170.216.18", 
            "AssociationOverview": {
                "DetailedStatus": "Failed", 
                "InstanceAssociationStatusAggregatedCount": {
                    "Failed": 1, 
                    "Success": 1
                }
            }, 
            "AssociationStatus": "Failed",  
            "PlatformVersion": "2", 
            "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", 
            "InstanceId": "i-00000000001", 
            "PlatformName": "Amazon Linux"
        }
    ]
}

 

 

 And you can find my props.conf below :

 

[my_test]
SHOULD_LINEMERGE = false
INDEXED_EXTRACTIONS = json
DATETIME_CONFIG = CURRENT
TRUNCATE = 999999
JSON_TRIM_BRACES_IN_ARRAY_NAMES = true
BREAK_ONLY_BEFORE = (\[\s+\{)
MUST_BREAK_AFTER = (\},|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_footer = s/\]\s+\}//g

 

 

 

Can you help me to find the write parsing please ?

Thank you.

Labels (2)
0 Karma

mah
Builder

Hi @richgalloway 

I want to break where I have : closed braces / comma / opened braces :

   "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal"
        }, 
        {
            "PlatformType": "Linux",  

Thanks a lot for your help !

 

 

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

You could try this.

| makeresults 
| eval _raw = "{
    \"InstanceInformationList\": [
        {
            \"Version\": false,
            \"PlatformName\": \"Amazon Linux\",
            \"ComputerName\": \"ip-10-170-216-17.eu-east-1.compute.internal\"
        },
        {
            \"PlatformType\": \"Linux\",
            \"IPAddress\": \"10.170.216.18\",
            \"AssociationOverview\": {
                \"DetailedStatus\": \"Failed\",
                \"InstanceAssociationStatusAggregatedCount\": {
                    \"Failed\": 1,
                    \"Success\": 1
                }
            },
            \"AssociationStatus\": \"Failed\",
            \"PlatformVersion\": \"2\",
            \"ComputerName\": \"ip-10-170-216-18.eu-east-1.compute.internal\",
            \"InstanceId\": \"i-00000000001\",
            \"PlatformName\": \"Amazon Linux\"
        }
    ]
}"
| spath output=Computer path=InstanceInformationList{}.ComputerName

Using above you could get mv field Computer where are those ComputerNames.

r. Ismo

0 Karma

mah
Builder

I want to break the json at indextime because mvexpand command is ressource consuming. 

I want  to separate the json in order to have 1 braces by event : 

One event :

 { "Version": false, "PlatformName": "Amazon Linux", "ComputerName": "ip-10-170-216-17.eu-east-1.compute.internal" }

another event :

{ "PlatformType": "Linux", "IPAddress": "10.170.216.18", "AssociationOverview": { "DetailedStatus": "Failed", "InstanceAssociationStatusAggregatedCount": { "Failed": 1, "Success": 1 } }, "AssociationStatus": "Failed", "PlatformVersion": "2", "ComputerName": "ip-10-170-216-18.eu-east-1.compute.internal", "InstanceId": "i-00000000001", "PlatformName": "Amazon Linux" }

Wherever there is this : }, {  split the json by using setting in props.conf

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

Are you really sure that you want break that JSON to several events on indexing time? If you do it then you cannot manage it as whole JSON instance later on?

If this is really what you want to do, you could try this

DATETIME_CONFIG=CURRENT
SHOULD_LINEMERGE=true
LINE_BREAKER=(\s+\[\s+\{)|(\s+\},\s+\{\s+)|(\}\s+\])
NO_BINARY_CHECK=true
TRUNCATE=0

Basically you must forget that this is JSON and manage it as normal text event.

I think that you must use separate transforms.conf to get rid of those "unnecessary" header and footer. Based on order which props + transforms are handled you cannot use SEDCMD in this case. 

r. Ismo

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where do you want to break the event?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...