Getting Data In

De-nesting JSON during indexing

clong_
Engager

I have the following event from GCP pubsub:

{ 
   attributes: {
   }

   data: { 
     insertIddbp95qcbup
     logNameorganizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access
     protoPayload: { [+]
     }

     receiveTimestamp2021-08-02T05:52:58.861079027Z
     resource: { [+]
     }

     severityNOTICE
     timestamp2021-08-02T04:01:48.076823Z
   }

   publish_time1627883579.307

}

 

Is there any way to use a forwarder to only send the contents of data{} to Splunk? I essentially want to strip off the outer parts of the JSON attributes{}, publishtime and have the event sent as the contents of the data{} field:"

{
"insertId": "dbp95qcbup",
"logName": "organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access",
"protoPayload": {},
"receiveTimestamp": "2021-08-02T05:52:58.861079027Z",
"resource": {},
"severity": "NOTICE",
"timestamp": "2021-08-02T04:01:48.076823Z"
}
Labels (2)
0 Karma

briqpayerik
Explorer

Hi @kamlesh_vaghela,

I also want to de-nest logs that have been pulled from GCP. have Splunk Cloud and as far as I know, I can't edit props.conf, is it possible to achieve this using the GUI in "Field transformations"?

0 Karma

isoutamo
SplunkTrust
SplunkTrust

You could/should setup a HF which do that before logs have sent to SC.

r. Ismo

briqpayerik
Explorer

Hi!

I followed your advice to setup a Heavy Forwarder and installed the Splunk Add-on for Google Cloud Platform. Then, I configured it to forward logs to Splunk Cloud.
I configured props.conf like in the original answer (on the HF) and can verify in the GUI that the changes has applied but the logs that are forwarded to SC have not been transformed / de-nested. I don't know where the problem lies.

Question 1: Should the source type in props.conf be [google:gcp:pubsub:message] ?
Question 2: Should props.conf be configured on the HF or SC?
Question 3: Since I realised I can make this configuration in the GUI (on SC or IDM would work?), do I even need the HF?

HF config:
Screenshot 2022-01-11 at 17.40.02.png

Logs on SC:
Screenshot 2022-01-11 at 17.49.48.png

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

When you are using TA/SA etc. it's usually best to use those same sourcetype names what has defined there. There could be additional TA/SA/Apps etc which are expecting those names. But if you want, you can choose what ever you like.

If you are doing ingestion time data management then it must be in the first full splunk instance in path from source to indexers. If only search time, then SC is the correct place. And if there is both then you needs it in both places. But read the TA's installation instructions which told how to install it in distributed environment! I'm not sure if you can install it by yourself int SC with Victoria experience or not? With GUI you could add those individual KOs, but I prefer to use TA as is if possible.

If I recall right this TA is using polling from GCP to get those events. This means that you are needing some place where it can do it. And there can be only one instance running at same time, otherwise you will get (at least partially) duplicate events as individual instances cannot know what other have already gotten. Also you have full control to separate HF instead of SC where you haven't any control to indexer layer. At least I prefer separate HF for those.

r. Imo

0 Karma

briqpayerik
Explorer

We were able to get everything working in the end! Not sure what the exact problem was but the regex was invalid, we used regex101 with the  python flavor to fix and build our regex.

0 Karma

SinghK
Builder

Isn't spath command working for you, can't think of anything else..

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@clong_ 

Can you please try this props.conf configuration ?

 

 

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json
SEDCMD-a=s/{ \"attributes\": \{.*\},\"data\":\s//g
SEDCMD-b=s/,\"publish_time\":\s\".*//g

 

 

 

My Sample Event:

 

 

{ "attributes": {},"data": { "insertId": "dbp95qcbup","logName": "organizations/xxxxxxx/logs/cloudaudit.googleapis.com%2Fdata_access","protoPayload" : { },"receiveTimestamp" : "2021-08-02T05:52:58.861079027Z","resource": {},"severity": "NOTICE","timestamp": "2021-08-02T04:01:48.076823Z"},"publish_time": "1627883579.307"}

 

 

Screenshot 2021-08-04 at 6.10.04 PM.png

 

KV 

 

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...