Knowledge Management

Setting up SQS based S3 input= Is it a requirement that SQS is updated via a subscription to an SNS topic?

Stokers_23
Explorer

Hi

I am running an splunk instance within my AWS account, and i'm trying to setup an Cloudtrail SQS based S3 imput. The cloud trail logs are stored in a bucket (auditlogs) in separate account, which I access via a switch role.

I have done the following however no data appears in index I have selected

  • Created an IAM policy with the required permissions
  • Created the required SQS Queue, granting permissions to the auditlogs bucket to post events.
  • Added an event notification on the S3 bucket to forward 'Object-created' events to my SQS Queue
  • Confirmed that the SQS Queue is receiving messages
  • Added a new input within the AWS Add on for splunk web, using my auto discovered IAM role
  • Requested for the input sends data to my aduit index.
  • Checked the logs on the splunk instance and found no errors, other issues.

Questions
- The documentation seems very unclear on the need to have an SNS topic in the middle here? Is it a requirement that SQS is updated via a subscription to an SNS topic. Specifically S3 > SNS > SQS > Splunk? Or would S3 > SQS > Splunk also work?

  • My auto discovered IAM role applied to the splunk EC2 instance is in a separate account to the S3 bucket i'm trying to import data from. Is this going to cause me issues - I assume this is the issue, but there

I would appreciate any guidance here!

Thanks

Labels (1)
Tags (1)

thambisetty
SplunkTrust
SplunkTrust

CURRENT APPLICATION

Splunk Add-on for AWS

Version: 6.2.0 Build: 1658820915

 

Simple Architecture:

thambisetty_0-1666155912165.png



Expected process by Splunk Add-on for AWS is as below:

1. Enable/configure cloud trail logs to s3 bucket and enable SNS topic 
2. create standard SQS queue ( This queue will be used as Dead Letter queue in SQS queue creation of 3rd step)
3. Create standard SQS queue with below configuration

Make sure your SQS queues have same configuration as below (except name, because your SQS will have different name)

thambisetty_0-1666154389016.png

DLQ Configuration for SQS created in step 3 : Choose queue created in 2 step. 

thambisetty_1-1666154470228.png

after creating the queue from step 3 . open queue created from step 3 and subscribe to SNS topic

thambisetty_2-1666154677836.png

Modify SQS created in step 3 Policy

 

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:*",
      "Resource": "<arn:aws:sqs:us-east-1:000000000000:this-sqs-queue>",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "<arn:aws:sns:us-east-1:000000000000:your-sns-topic"
        }
      }
    }

 


4. Above enabled SNS topic will be subscribed by SQS

Below is the SNS Access policy : easiest way to get this policy created is to create SNS topic while enabling SNS in cloud trail log setup. 

 

 

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:GetTopicAttributes",
        "SNS:SetTopicAttributes",
        "SNS:AddPermission",
        "SNS:RemovePermission",
        "SNS:DeleteTopic",
        "SNS:Subscribe",
        "SNS:ListSubscriptionsByTopic",
        "SNS:Publish"
      ],
      "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "<account_id>"
        }
      }
    },
    {
      "Sid": "AWSCloudTrailSNSPolicy20150319",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "<arn:aws:cloudtrail:us-east-1:0000000000:trail/cloudtrail-events>"
        }
      }
    }
  ]
}

 


5. Create IAM Policy ( Added permissions based on my best knowledge and keeping least privilege in mind)  using below: Note: don't forget to change SQS and S3 ARNs

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:GetQueueUrl",
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketTagging",
                "sqs:ReceiveMessage",
                "s3:GetBucketLogging",
                "sqs:SendMessage",
                "sqs:GetQueueAttributes",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetObject",
                "s3:GetBucketCORS",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "<arn:aws:sqs:us-east-1:00000000:sqs-name>",
                "<arn:aws:s3:::bucket_name>",
                "<arn:aws:s3:::bucket_name>/*>"
            ]
        }
    ]
}​

 

6. Create a new role and choose above IAM policy while creating.
7. create a user

8. In the role trust relationships add below policy to allow user to assume this role. 

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<arn:aws:iam::00000000000:user/created_in_step7>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

 

Note: don't try to copy and paste the above policy. some characters are will not be copied properly. especially IAM policy where it gives s3 permission to resource object "buket_arn/*"

————————————
If this helps, give a like below.
Tags (1)
0 Karma

wagnerlucena
Explorer

You need to follow the second option.

2> aws cloudtrail/config > S3 bucket > Event notification trigger as SNS > SQS subscription > Splunk

Enable the cloudtrail sending notification to SNS

alt text

Create SQS and subscribe the SNS to it

In splunk, create the cloudtrail input and choose the SQS that was created in the last step.

alt text

wagnerlucena
Explorer

don't forget upvote if the answer helped you.

0 Karma

meet_vadaria
Engager

I am also confused at this. Document is very unclear. I tried to google it. went through few blogs and videos. All are different. there are 3 ways to do this as far as I know,
1> aws cloudtrail/config > SNS notification enabled at source with S3 bucket > SQS subscription > Splunk (in this case there is a no event notification trigger in S3 bucket)

2> aws cloudtrail/config > S3 bucket > Event notification trigger as SNS > SQS subscription > Splunk

3> aws cloudtrail/config > S3 bucket > Event notification trigger as SQS > Splunk

I am confused which on to follow. So far, I have tried 3rd option. it's working but I am seeing couple of errors "Unable to parse message" from both aws cloudtrail and aws config.

Will be great if someone can explain!

0 Karma

vgollapudi
Communicator

The option 2 is the best one to go for.

0 Karma

om
Splunk Employee
Splunk Employee

What is the difference between Option 2. and Option 3 ? Either way, doesn't the SQS queue get the same event ?

0 Karma

thambisetty
SplunkTrust
SplunkTrust

SQS notification can be created from both the ways. But Splunk add-on for aws is designed in way that has dependency to validate signature of the object.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...