Knowledge Management

Setting up SQS based S3 input= Is it a requirement that SQS is updated via a subscription to an SNS topic?

Stokers_23
Explorer

Hi

I am running an splunk instance within my AWS account, and i'm trying to setup an Cloudtrail SQS based S3 imput. The cloud trail logs are stored in a bucket (auditlogs) in separate account, which I access via a switch role.

I have done the following however no data appears in index I have selected

  • Created an IAM policy with the required permissions
  • Created the required SQS Queue, granting permissions to the auditlogs bucket to post events.
  • Added an event notification on the S3 bucket to forward 'Object-created' events to my SQS Queue
  • Confirmed that the SQS Queue is receiving messages
  • Added a new input within the AWS Add on for splunk web, using my auto discovered IAM role
  • Requested for the input sends data to my aduit index.
  • Checked the logs on the splunk instance and found no errors, other issues.

Questions
- The documentation seems very unclear on the need to have an SNS topic in the middle here? Is it a requirement that SQS is updated via a subscription to an SNS topic. Specifically S3 > SNS > SQS > Splunk? Or would S3 > SQS > Splunk also work?

  • My auto discovered IAM role applied to the splunk EC2 instance is in a separate account to the S3 bucket i'm trying to import data from. Is this going to cause me issues - I assume this is the issue, but there

I would appreciate any guidance here!

Thanks

Labels (1)
Tags (1)

thambisetty
Super Champion

CURRENT APPLICATION

Splunk Add-on for AWS

Version: 6.2.0 Build: 1658820915

 

Simple Architecture:

thambisetty_0-1666155912165.png



Expected process by Splunk Add-on for AWS is as below:

1. Enable/configure cloud trail logs to s3 bucket and enable SNS topic  2. create standard SQS queue ( This queue will be used as Dead Letter queue) 3. Create standard SQS queue with below configuration

thambisetty_0-1666154389016.png

 

DLQ Configuration: Choose queue created in 2 step. 

thambisetty_1-1666154470228.png

after creating the queue. open queue and subscribe to SNS topic

thambisetty_2-1666154677836.png

 

Modify SQS Policy

 

 

 

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:*",
      "Resource": "<arn:aws:sqs:us-east-1:000000000000:this-sqs-queue>",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "<arn:aws:sns:us-east-1:000000000000:your-sns-topic"
        }
      }
    }

 

 

 

 


4. Above enabled SNS topic will be subscribed by SQS

Below is the SNS Access policy : easiest way to get this policy created is to create SNS topic while enabling SNS in cloud trail log setup. 

 

 

 

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:GetTopicAttributes",
        "SNS:SetTopicAttributes",
        "SNS:AddPermission",
        "SNS:RemovePermission",
        "SNS:DeleteTopic",
        "SNS:Subscribe",
        "SNS:ListSubscriptionsByTopic",
        "SNS:Publish"
      ],
      "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "<account_id>"
        }
      }
    },
    {
      "Sid": "AWSCloudTrailSNSPolicy20150319",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "<arn:aws:sns:us-east-1:0000000000:sns-topic>",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "<arn:aws:cloudtrail:us-east-1:0000000000:trail/cloudtrail-events>"
        }
      }
    }
  ]
}

 

 


5. Create IAM Policy ( Added permissions based on my best knowledge and keeping least privilege in my mind)  using below: Note: don't forget to change SQS and S3 ARNs

 

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:GetQueueUrl",
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketTagging",
                "sqs:ReceiveMessage",
                "s3:GetBucketLogging",
                "sqs:SendMessage",
                "sqs:GetQueueAttributes",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetObject",
                "s3:GetBucketCORS",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "<arn:aws:sqs:us-east-1:00000000:sqs-name>",
                "<arn:aws:s3:::bucket_name>",
                "<arn:aws:s3:::bucket_name>/*>"
            ]
        }
    ]
}​

 

 

 

6. Create a new role and choose above IAM policy while creating.
7. create a user

8. In the role trust relationships add below policy to allow user to assume this role. 

 

 

 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<arn:aws:iam::00000000000:user/created_in_step7>"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

 

 

Note: don't try to copy and paste the above policy. some characters are will not be copied properly. especially IAM policy where it gives s3 permission to resource object "buket_arn/*"


 

————————————
If this helps, give a like below.
Tags (1)
0 Karma

wagnerlucena
Explorer

You need to follow the second option.

2> aws cloudtrail/config > S3 bucket > Event notification trigger as SNS > SQS subscription > Splunk

Enable the cloudtrail sending notification to SNS

alt text

Create SQS and subscribe the SNS to it

In splunk, create the cloudtrail input and choose the SQS that was created in the last step.

alt text

wagnerlucena
Explorer

don't forget upvote if the answer helped you.

0 Karma

meet_vadaria
Engager

I am also confused at this. Document is very unclear. I tried to google it. went through few blogs and videos. All are different. there are 3 ways to do this as far as I know,
1> aws cloudtrail/config > SNS notification enabled at source with S3 bucket > SQS subscription > Splunk (in this case there is a no event notification trigger in S3 bucket)

2> aws cloudtrail/config > S3 bucket > Event notification trigger as SNS > SQS subscription > Splunk

3> aws cloudtrail/config > S3 bucket > Event notification trigger as SQS > Splunk

I am confused which on to follow. So far, I have tried 3rd option. it's working but I am seeing couple of errors "Unable to parse message" from both aws cloudtrail and aws config.

Will be great if someone can explain!

0 Karma

vgollapudi
Communicator

The option 2 is the best one to go for.

0 Karma

om
Splunk Employee
Splunk Employee

What is the difference between Option 2. and Option 3 ? Either way, doesn't the SQS queue get the same event ?

0 Karma

thambisetty
Super Champion

SQS notification can be created from both the ways. But Splunk add-on for aws is designed in way that has dependency to validate signature of the object.

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...