For the past 2 days I'm trying to figure something out. I'll try to be clear as possible and hopefully that someone can guide me or explain why this is working like this. I'm trying to index a CSV file stored in S3, but unfortunately the sourcetype aws:s3:csv is not indexing the file "properly" (meaning it is not extracting any fields - check left screenshot from the attached file). I've modified the sourcetype aws:s3:csv (under the Splunk Addon for AWS application) and configured it exactly like the default CSV sourcetype (under system/default/proprs.conf). After doing this if I index a file manually via "Settings/Add data" it is being indexed properly (fields are being extracted), but if the very same file is indexed by the app Splunk Addon for AWS,again  configured with the same sourcetype, there are no extracted fields. Check attached screenshot for reference. I've also tried to add other different configurations to the not-modified aws:s3:csv sourcetype like INDEXED_EXTRACTIONS = CSV; HEADER_FIELD_LINE_NUMBER = 1; FIELD_NAMES = field1,field2,field3 and various other configurations in props.conf (under Splunk Addon for AWS) but without success. The only "workaround" is if I use REPORT-extract_fields in props.conf for that sourcetype and in transforms.conf configure it, but this is not ideal. Additionally I've set the sourcetype to csv  (default Splunk sourcetype) in the inputs.conf but this also seems to not work. Splunk 9.2.1 Splunk Add-on for AWS 7.7.0 Similar questions without proper answer: https://community.splunk.com/t5/All-Apps-and-Add-ons/Splunk-Add-on-for-Amazon-Web-Services-How-to-get-a-CSV-file/m-p/131725 https://community.splunk.com/t5/Splunk-Enterprise/Splunk-Add-on-for-AWS-Ingesting-csv-files-and-he-fields-are-not/td-p/656923 https://community.splunk.com/t5/All-Apps-and-Add-ons/S3-bucket-with-CSV-files-not-extracting-fields-at-index-time/m-p/458671 https://community.splunk.com/t5/Getting-Data-In/No-fields-or-timestamps-extracted-when-indexing-TSV-from-S3/m-p/660436 https://community.splunk.com/t5/Getting-Data-In/Why-is-CSV-data-not-getting-parsed-while-being-monitored-on/td-p/275515 ... View more

I've searched quite some time, but I'm not able to find why Splunk is not recognizing a nested JSON. Here's how my data/events looks in raw text ( path of the data is SES->SNS->Lambda->HEC)     {"Records":[{"EventSource":"aws:sns","EventVersion":"1.0","EventSubscriptionArn":"arn:aws:sns:eu-north-1:doesntmatter","Sns":{"Type":"Notification","MessageId":"87b93315-f1f6-56f8-83dc-6b099eb5e18e","TopicArn":"arn:aws:sns:eu-north-1:doesntmatter","Subject":null,"Message":"{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2020-11-04T08:57:37.646Z\",\"source\":\"email@email.com\",\"sourceArn\":\"arn:aws:ses:eu-north-1:MYaccountID:identity/email@email.com\",\"sourceIp\":\"X.X.X.X\",\"sendingAccountId\":\"MYaccountID\",\"messageId\":\"011001759279ce6e-67642459-31fa-4f4b-b852-315ea7e8d284-000000\",\"destination\":[\"email@email.com\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"Received\",\"value\":\"from ip-X.X.X.X.eu-north-1.compute.internal (ec2-X.X.X.X.eu-north-1.compute.amazonaws.com [X.X.X.X]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-d-090KRTZ85) id RANDOM for email@email.com; Wed, 04 Nov 2020 08:57:37 +0000 (UTC)\"},{\"name\":\"Content-Type\",\"value\":\"multipart/mixed; boundary=\\\"===============digits==\\\"\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"Subject\",\"value\":\"to me from NEW NEW\"},{\"name\":\"To\",\"value\":\"email@email.com\"},{\"name\":\"From\",\"value\":\"email@email.com\"},{\"name\":\"Date\",\"value\":\"Wed, 04 Nov 2020 08:57:37 +0000\"},{\"name\":\"X-Priority\",\"value\":\"3\"},{\"name\":\"X-Splunk-SID\",\"value\":\"digits.2\"},{\"name\":\"X-Splunk-ServerName\",\"value\":\"ip-X.X.X.X.eu-north-1.compute.internal\"},{\"name\":\"X-Splunk-Version\",\"value\":\"8.1.0\"},{\"name\":\"X-Splunk-Build\",\"value\":\"f57c09e87251\"}],\"commonHeaders\":{\"from\":[\"email@email.com\"],\"date\":\"Wed, 04 Nov 2020 08:57:37 +0000\",\"to\":[\"email@email.com\"],\"subject\":\"to me from NEW NEW\"}},\"delivery\":{\"timestamp\":\"2020-11-04T08:57:39.153Z\",\"processingTimeMillis\":1507,\"recipients\":[\"email@email.com\"],\"smtpResponse\":\"250 2.6.0 <digits@eu-north-1.amazonses.com> [InternalId=digits, Hostname=random hostname] 12107 bytes in 0.057, 206.415 KB/sec Queued mail for delivery\",\"remoteMtaIp\":\"Y.Y.Y.Y\",\"reportingMTA\":\"e240-9.smtp-out.eu-north-1.amazonses.com\"}}","Timestamp":"2020-11-04T08:57:39.198Z","SignatureVersion":"1","Signature":"SIGNATURE","SigningCertUrl":"https://sns.eu-north-1.amazonaws.com/SimpleNotificationService-PEM.pem","UnsubscribeUrl":"https://sns.eu-north-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-north-1:doesntmatter","MessageAttributes":{}}}]}     I'm sending the data to :8088/collector  . Here's the screenshot of how the data looks like in syntax:   I would like to extract some fields from "Message" which is a valid JSON, however I'm not able to do it since Splunk is not recognizing it as a JSON and my knowledge about filtering using rex is below 0  😐 . I believe it has something to do with the backslashes, but when I use to remove them:     SEDCMD-replace_backslash = s/\\//g     in my props.conf, Splunk stops to recognize the whole event as json and it's not formatting it (showing as raw when I do a search). My props.conf looks like:       [ses_json_new] category = Custom pulldown_type = 1     If I add to props.conf     KV_MODE = json       Nothing happens, nor with:     INDEXED_EXTRACTIONS = json     Appreciate your help and please excuse me for my nescience. Thank you. ... View more