Getting Data In

De-nesting JSON during indexing

clong_
Engager

I have the following event from GCP pubsub:

{ 
   attributes: {
   }

   data: { 
     insertIddbp95qcbup
     logNameorganizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access
     protoPayload: { [+]
     }

     receiveTimestamp2021-08-02T05:52:58.861079027Z
     resource: { [+]
     }

     severityNOTICE
     timestamp2021-08-02T04:01:48.076823Z
   }

   publish_time1627883579.307

}

 

Is there any way to use a forwarder to only send the contents of data{} to Splunk? I essentially want to strip off the outer parts of the JSON attributes{}, publishtime and have the event sent as the contents of the data{} field:"

{
"insertId": "dbp95qcbup",
"logName": "organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access",
"protoPayload": {},
"receiveTimestamp": "2021-08-02T05:52:58.861079027Z",
"resource": {},
"severity": "NOTICE",
"timestamp": "2021-08-02T04:01:48.076823Z"
}
Labels (2)
0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@clong_ 

Can you please try this props.conf configuration ?

 

 

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
SEDCMD-a=s/{ \"attributes\": \{.*\},\"data\":\s//g
SEDCMD-b=s/,\"publish_time\":\s\".*//g

 

 

 

My Sample Event:

 

 

{ "attributes": {},"data": { "insertId": "dbp95qcbup","logName": "organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access","protoPayload" : { },"receiveTimestamp" : "2021-08-02T05:52:58.861079027Z","resource": {},"severity": "NOTICE","timestamp": "2021-08-02T04:01:48.076823Z"},"publish_time": "1627883579.307"}

 

 

Screenshot 2021-08-04 at 6.10.04 PM.png

 

KV 

 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!