Getting Data In

Splunk not recognizing nested json

dstoev
Explorer

I've searched quite some time, but I'm not able to find why Splunk is not recognizing a nested JSON.

Here's how my data/events looks in raw text ( path of the data is SES->SNS->Lambda->HEC)

 

 

{"Records":[{"EventSource":"aws:sns","EventVersion":"1.0","EventSubscriptionArn":"arn:aws:sns:eu-north-1:doesntmatter","Sns":{"Type":"Notification","MessageId":"87b93315-f1f6-56f8-83dc-6b099eb5e18e","TopicArn":"arn:aws:sns:eu-north-1:doesntmatter","Subject":null,"Message":"{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2020-11-04T08:57:37.646Z\",\"source\":\"email@email.com\",\"sourceArn\":\"arn:aws:ses:eu-north-1:MYaccountID:identity/email@email.com\",\"sourceIp\":\"X.X.X.X\",\"sendingAccountId\":\"MYaccountID\",\"messageId\":\"011001759279ce6e-67642459-31fa-4f4b-b852-315ea7e8d284-000000\",\"destination\":[\"email@email.com\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"Received\",\"value\":\"from ip-X.X.X.X.eu-north-1.compute.internal (ec2-X.X.X.X.eu-north-1.compute.amazonaws.com [X.X.X.X]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-d-090KRTZ85) id RANDOM for email@email.com; Wed, 04 Nov 2020 08:57:37 +0000 (UTC)\"},{\"name\":\"Content-Type\",\"value\":\"multipart/mixed; boundary=\\\"===============digits==\\\"\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"Subject\",\"value\":\"to me from NEW NEW\"},{\"name\":\"To\",\"value\":\"email@email.com\"},{\"name\":\"From\",\"value\":\"email@email.com\"},{\"name\":\"Date\",\"value\":\"Wed, 04 Nov 2020 08:57:37 +0000\"},{\"name\":\"X-Priority\",\"value\":\"3\"},{\"name\":\"X-Splunk-SID\",\"value\":\"digits.2\"},{\"name\":\"X-Splunk-ServerName\",\"value\":\"ip-X.X.X.X.eu-north-1.compute.internal\"},{\"name\":\"X-Splunk-Version\",\"value\":\"8.1.0\"},{\"name\":\"X-Splunk-Build\",\"value\":\"f57c09e87251\"}],\"commonHeaders\":{\"from\":[\"email@email.com\"],\"date\":\"Wed, 04 Nov 2020 08:57:37 +0000\",\"to\":[\"email@email.com\"],\"subject\":\"to me from NEW NEW\"}},\"delivery\":{\"timestamp\":\"2020-11-04T08:57:39.153Z\",\"processingTimeMillis\":1507,\"recipients\":[\"email@email.com\"],\"smtpResponse\":\"250 2.6.0 <digits@eu-north-1.amazonses.com> [InternalId=digits, Hostname=random hostname] 12107 bytes in 0.057, 206.415 KB/sec Queued mail for delivery\",\"remoteMtaIp\":\"Y.Y.Y.Y\",\"reportingMTA\":\"e240-9.smtp-out.eu-north-1.amazonses.com\"}}","Timestamp":"2020-11-04T08:57:39.198Z","SignatureVersion":"1","Signature":"SIGNATURE","SigningCertUrl":"https://sns.eu-north-1.amazonaws.com/SimpleNotificationService-PEM.pem","UnsubscribeUrl":"https://sns.eu-north-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-north-1:doesntmatter","MessageAttributes":{}}}]}

 

 

I'm sending the data to :8088/collector  . Here's the screenshot of how the data looks like in syntax:

splunk_nested_json.png

 

I would like to extract some fields from "Message" which is a valid JSON, however I'm not able to do it since Splunk is not recognizing it as a JSON and my knowledge about filtering using rex is below 0  😐 .

I believe it has something to do with the backslashes, but when I use to remove them:

 

 

SEDCMD-replace_backslash = s/\\//g 

 

 


in my props.conf, Splunk stops to recognize the whole event as json and it's not formatting it (showing as raw when I do a search).

My props.conf looks like:

 

 

 

[ses_json_new]
category = Custom
pulldown_type = 1

 

 


If I add to props.conf

 

 

KV_MODE = json

 

 

 

Nothing happens, nor with:

 

 

INDEXED_EXTRACTIONS = json

 

 

Appreciate your help and please excuse me for my nescience.

Thank you.

0 Karma
Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...