Can we integrate the Splunk App for AWS in our Splunk Enterprise environment?

[aaguirr1]

Hi all,

We have deployed Splunk Enterprise on premise to manage the logs of different IT assets. At present, we are going to use AWS services. (EC2, VPC, RDS, S3, CloudConfig, etc.)

I have the next question:

1) Can we integrate Splunk App for AWS in our Splunk Enterprise? I would like to have the same dashboard to manage the logs of AWS console and IT assets.

2) We have 3 accounts of AWS (Prod, Dev and R&D). Do we need install an indexer in each account? Overall 3 indexers.

If you need more information, please let me know.

Thanks in advance.

Regards,
Arsenio

[Jeremiah]

You can deploy the Splunk app for AWS in a several different ways. You should think about the app as having 3 functions: data collection, storage, and UI. You can run all or some of those functions in your data center, in AWS or in a hybrid deployment.

For example, you could run an entirely on-prem solution, with a search head that runs the AWS app (provides the UI capability), but also collects data from AWS (data collection), and forwards that data to an indexer (the storage). Or you could deploy a search head in your corporate data center with the AWS app installed, and then perform the collection on a forwarder running the AWS app in AWS and send data to an indexer also running in AWS.

You might want an indexer in each account (VPC) if you are running forwarders on your EC2 instances. If not, there isn't any direct connection between your account/VPCs and the Splunk app for AWS. The app just needs connectivity to AWS endpoints which are accessible over the internet.

[n6BXGybt]

I'm not sure about question 1) however for question 2) I suspect that you don't need an indexer for the AWS specific logs as there are no Sizing and Performance considerations.

Reference: http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/Sizingandcost

In my case, I have four AWS accounts (DEV, QA, UAT, and PROD). Since assume role is not supported, I have an IAM user account in each AWS account - DEV-SPLUNKAPP-USER for example. This user has in inline policy configured as below and per the Splunk documentation for using one policy for all inputs:

Reference: http://docs.splunk.com/Documentation/AWS/4.2.1/Installation/ConfigureyourAWSpermissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueAttributes",
                "sqs:ListQueues",
                "sqs:ReceiveMessage",
                "sqs:GetQueueUrl",
                "sqs:SendMessage",
                "sqs:DeleteMessage",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets",
                "config:DeliverConfigSnapshot",
                "config:DescribeConfigRules",
                "config:DescribeConfigRuleEvaluationStatus",
                "config:GetComplianceDetailsByConfigRule",
                "config:GetComplianceSummaryByConfigRule",
                "config:DescribeDeliveryChannels",
                "cloudtrail:DescribeTrails",
                "autoscaling:Describe*",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "sns:Get*",
                "sns:List*",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "ec2:DescribeInstances",
                "ec2:DescribeReservedInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeRegions",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "rds:DescribeDBInstances",
                "cloudfront:ListDistributions",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeInstanceHealth",
                "inspector:Describe*",
                "inspector:List*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

In my case I also created a separate index in Splunk (Settings --> Indexes) for each AWS service. (eg. config_logs, cloudwatch_logs, etc.)