topic Re: De-nesting JSON during indexing in Getting Data In

De-nesting JSON during indexing

clong_ — Mon, 02 Aug 2021 22:44:07 GMT

I have the following event from GCP pubsub:

{
   attributes: {
   }
   data: {
     insertId: dbp95qcbup
     logName: organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access
     protoPayload: { [+]
     }
     receiveTimestamp: 2021-08-02T05:52:58.861079027Z
     resource: { [+]
     }
     severity: NOTICE
     timestamp: 2021-08-02T04:01:48.076823Z
   }
   publish_time: 1627883579.307
}

Is there any way to use a forwarder to only send the contents of data{} to Splunk? I essentially want to strip off the outer parts of the JSON attributes{}, publishtime and have the event sent as the contents of the data{} field:"

{

"insertId": "dbp95qcbup",

"logName": "organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access",

"protoPayload": {},

"receiveTimestamp": "2021-08-02T05:52:58.861079027Z",

"resource": {},

"severity": "NOTICE",

"timestamp": "2021-08-02T04:01:48.076823Z"

}

Re: De-nesting JSON during indexing

kamlesh_vaghela — Wed, 04 Aug 2021 12:40:19 GMT

@clong_

Can you please try this props.conf configuration ?

[YOUR_SOURCETYPE] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 INDEXED_EXTRACTIONS=json SEDCMD-a=s/{ \"attributes\": \{.*\},\"data\":\s//g SEDCMD-b=s/,\"publish_time\":\s\".*//g

My Sample Event:

{ "attributes": {},"data": { "insertId": "dbp95qcbup","logName": "organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access","protoPayload" : { },"receiveTimestamp" : "2021-08-02T05:52:58.861079027Z","resource": {},"severity": "NOTICE","timestamp": "2021-08-02T04:01:48.076823Z"},"publish_time": "1627883579.307"}

Re: De-nesting JSON during indexing

briqpayerik — Fri, 07 Jan 2022 10:41:42 GMT

Hi @kamlesh_vaghela,

I also want to de-nest logs that have been pulled from GCP. have Splunk Cloud and as far as I know, I can't edit props.conf, is it possible to achieve this using the GUI in "Field transformations"?

Re: De-nesting JSON during indexing

SinghK — Fri, 07 Jan 2022 10:49:35 GMT

Isn't spath command working for you, can't think of anything else..

Re: De-nesting JSON during indexing

isoutamo — Fri, 07 Jan 2022 16:46:14 GMT

You could/should setup a HF which do that before logs have sent to SC.

r. Ismo

Re: De-nesting JSON during indexing

briqpayerik — Tue, 11 Jan 2022 16:56:20 GMT

Hi!

I followed your advice to setup a Heavy Forwarder and installed the Splunk Add-on for Google Cloud Platform. Then, I configured it to forward logs to Splunk Cloud.
I configured props.conf like in the original answer (on the HF) and can verify in the GUI that the changes has applied but the logs that are forwarded to SC have not been transformed / de-nested. I don't know where the problem lies.

Question 1: Should the source type in props.conf be [google:gcp:pubsub:message] ?
Question 2: Should props.conf be configured on the HF or SC?
Question 3: Since I realised I can make this configuration in the GUI (on SC or IDM would work?), do I even need the HF?

HF config:

Logs on SC:

Re: De-nesting JSON during indexing

briqpayerik — Fri, 14 Jan 2022 10:10:38 GMT

We were able to get everything working in the end! Not sure what the exact problem was but the regex was invalid, we used regex101 with the python flavor to fix and build our regex.

Re: De-nesting JSON during indexing

isoutamo — Fri, 14 Jan 2022 10:17:16 GMT

When you are using TA/SA etc. it's usually best to use those same sourcetype names what has defined there. There could be additional TA/SA/Apps etc which are expecting those names. But if you want, you can choose what ever you like.

If you are doing ingestion time data management then it must be in the first full splunk instance in path from source to indexers. If only search time, then SC is the correct place. And if there is both then you needs it in both places. But read the TA's installation instructions which told how to install it in distributed environment! I'm not sure if you can install it by yourself int SC with Victoria experience or not? With GUI you could add those individual KOs, but I prefer to use TA as is if possible.

If I recall right this TA is using polling from GCP to get those events. This means that you are needing some place where it can do it. And there can be only one instance running at same time, otherwise you will get (at least partially) duplicate events as individual instances cannot know what other have already gotten. Also you have full control to separate HF instead of SC where you haven't any control to indexer layer. At least I prefer separate HF for those.

r. Imo