topic Re: Setting up SQS based S3 input= Is it a requirement that SQS is updated via a subscription to an SNS topic? in Knowledge Management

Setting up SQS based S3 input= Is it a requirement that SQS is updated via a subscription to an SNS topic?

Stokers_23 — Wed, 19 Oct 2022 14:02:11 GMT

I am running an splunk instance within my AWS account, and i'm trying to setup an Cloudtrail SQS based S3 imput. The cloud trail logs are stored in a bucket (auditlogs) in separate account, which I access via a switch role.

I have done the following however no data appears in index I have selected

Created an IAM policy with the required permissions
Created the required SQS Queue, granting permissions to the auditlogs bucket to post events.
Added an event notification on the S3 bucket to forward 'Object-created' events to my SQS Queue
Confirmed that the SQS Queue is receiving messages
Added a new input within the AWS Add on for splunk web, using my auto discovered IAM role
Requested for the input sends data to my aduit index.
Checked the logs on the splunk instance and found no errors, other issues.

Questions
- The documentation seems very unclear on the need to have an SNS topic in the middle here? Is it a requirement that SQS is updated via a subscription to an SNS topic. Specifically S3 > SNS > SQS > Splunk? Or would S3 > SQS > Splunk also work?

My auto discovered IAM role applied to the splunk EC2 instance is in a separate account to the S3 bucket i'm trying to import data from. Is this going to cause me issues - I assume this is the issue, but there

I would appreciate any guidance here!

Thanks

Re: Setting up SQS based S3 input!

meet_vadaria — Mon, 19 Nov 2018 18:50:58 GMT

I am also confused at this. Document is very unclear. I tried to google it. went through few blogs and videos. All are different. there are 3 ways to do this as far as I know,
1> aws cloudtrail/config > SNS notification enabled at source with S3 bucket > SQS subscription > Splunk (in this case there is a no event notification trigger in S3 bucket)

2> aws cloudtrail/config > S3 bucket > Event notification trigger as SNS > SQS subscription > Splunk

3> aws cloudtrail/config > S3 bucket > Event notification trigger as SQS > Splunk

I am confused which on to follow. So far, I have tried 3rd option. it's working but I am seeing couple of errors "Unable to parse message" from both aws cloudtrail and aws config.

Will be great if someone can explain!

Re: Setting up SQS based S3 input!

wagnerlucena — Thu, 22 Nov 2018 20:51:05 GMT

You need to follow the second option.

2> aws cloudtrail/config > S3 bucket > Event notification trigger as SNS > SQS subscription > Splunk

Enable the cloudtrail sending notification to SNS

Create SQS and subscribe the SNS to it

In splunk, create the cloudtrail input and choose the SQS that was created in the last step.

Re: Setting up SQS based S3 input!

wagnerlucena — Sat, 24 Nov 2018 14:22:57 GMT

don't forget upvote if the answer helped you.

Re: Setting up SQS based S3 input!

vgollapudi — Mon, 15 Apr 2019 21:39:32 GMT

The option 2 is the best one to go for.

Re: Setting up SQS based S3 input!

om — Thu, 01 Apr 2021 01:27:28 GMT

What is the difference between Option 2. and Option 3 ? Either way, doesn't the SQS queue get the same event ?

Re: Setting up SQS based S3 input= Is it a requirement that SQS is updated via a subscription to an SNS topic?

thambisetty — Fri, 17 Mar 2023 04:47:50 GMT

CURRENT APPLICATION

Splunk Add-on for AWS

Version: 6.2.0 Build: 1658820915

Simple Architecture:

Expected process by Splunk Add-on for AWS is as below:

1. Enable/configure cloud trail logs to s3 bucket and enable SNS topic
2. create standard SQS queue ( This queue will be used as Dead Letter queue in SQS queue creation of 3rd step)
3. Create standard SQS queue with below configuration

Make sure your SQS queues have same configuration as below (except name, because your SQS will have different name)

DLQ Configuration for SQS created in step 3 : Choose queue created in 2 step.

after creating the queue from step 3 . open queue created from step 3 and subscribe to SNS topic

Modify SQS created in step 3 Policy

{ "Version": "2008-10-17", "Id": "__default_policy_ID", "Statement": [ { "Sid": "__owner_statement", "Effect": "Allow", "Principal": "*", "Action": "SQS:*", "Resource": "<arn:aws:sqs:us-east-1:000000000000:this-sqs-queue>", "Condition": { "ArnEquals": { "aws:SourceArn": "<arn:aws:sns:us-east-1:000000000000:your-sns-topic" } } }

4. Above enabled SNS topic will be subscribed by SQS

Below is the SNS Access policy : easiest way to get this policy created is to create SNS topic while enabling SNS in cloud trail log setup.

{ "Version": "2008-10-17", "Id": "__default_policy_ID", "Statement": [ { "Sid": "__default_statement_ID", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "SNS:GetTopicAttributes", "SNS:SetTopicAttributes", "SNS:AddPermission", "SNS:RemovePermission", "SNS:DeleteTopic", "SNS:Subscribe", "SNS:ListSubscriptionsByTopic", "SNS:Publish" ], "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>", "Condition": { "StringEquals": { "AWS:SourceOwner": "<account_id>" } } }, { "Sid": "AWSCloudTrailSNSPolicy20150319", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "SNS:Publish", "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>", "Condition": { "StringEquals": { "AWS:SourceArn": "<arn:aws:cloudtrail:us-east-1:0000000000:trail/cloudtrail-events>" } } } ] }

5. Create IAM Policy ( Added permissions based on my best knowledge and keeping least privilege in mind) using below: Note: don't forget to change SQS and S3 ARNs

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "sqs:ListQueues", "s3:ListAllMyBuckets" ], "Resource": "*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "sqs:DeleteMessage", "sqs:GetQueueUrl", "s3:GetLifecycleConfiguration", "s3:GetBucketTagging", "sqs:ReceiveMessage", "s3:GetBucketLogging", "sqs:SendMessage", "sqs:GetQueueAttributes", "s3:ListBucket", "s3:GetAccelerateConfiguration", "s3:GetObject", "s3:GetBucketCORS", "s3:GetBucketLocation" ], "Resource": [ "<arn:aws:sqs:us-east-1:00000000:sqs-name>", "<arn:aws:s3:::bucket_name>", "<arn:aws:s3:::bucket_name>/*>" ] } ] }

6. Create a new role and choose above IAM policy while creating.
7. create a user

8. In the role trust relationships add below policy to allow user to assume this role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "<arn:aws:iam::00000000000:user/created_in_step7>" }, "Action": "sts:AssumeRole", "Condition": {} } ] }

Note: don't try to copy and paste the above policy. some characters are will not be copied properly. especially IAM policy where it gives s3 permission to resource object "buket_arn/*"

Re: Setting up SQS based S3 input!

thambisetty — Wed, 19 Oct 2022 05:27:56 GMT

SQS notification can be created from both the ways. But Splunk add-on for aws is designed in way that has dependency to validate signature of the object.