All Apps and Add-ons

Pull Logs from AWS CloudWatch to On-Prem Splunk Environment

kutubjt1
New Member

Whats the cleanest way to Pull Logs from AWS CloudWatch into On-Prem Splunk Environment.

"Pull" is the keyword here, we cannot do "push" to an HEC due to other achitectural constraints.

Also, the AWS environment is based on serverless architecture, so we cannot install a Heavy Forwarder within the AWS Environment.

Labels (1)
Tags (1)
0 Karma
1 Solution

pappjr
Path Finder

Hi @kutubjt1,

The best way to pull AWS CloudWatch logs into Splunk is to use the free Splunk Add-on for AWS. This app uses AWS account credentials to pull data from AWS APIs. All configuration can be done through the UI. In a distributed environment, I would recommend you install this on a "heavy forwarder" instance or similar.

The Splunk Add-on for AWS also includes other inputs for pulling data from your AWS account(s) - including VPC flow logs, S3 buckets, config, billing reports, instance metadata, etc.

I highly recommend you also install the Splunk App for AWS which provides out of the box visualizations for common use cases.

Hope that helps!

View solution in original post

0 Karma

pappjr
Path Finder

Hi @kutubjt1,

The best way to pull AWS CloudWatch logs into Splunk is to use the free Splunk Add-on for AWS. This app uses AWS account credentials to pull data from AWS APIs. All configuration can be done through the UI. In a distributed environment, I would recommend you install this on a "heavy forwarder" instance or similar.

The Splunk Add-on for AWS also includes other inputs for pulling data from your AWS account(s) - including VPC flow logs, S3 buckets, config, billing reports, instance metadata, etc.

I highly recommend you also install the Splunk App for AWS which provides out of the box visualizations for common use cases.

Hope that helps!

0 Karma

surajdevops18
Observer

Hello ,

We have Splunk Enterprise server on and installed the Splunk App for AWS and Splunk Add-on for AWS.

Configure AWS account details in Splunk configurations with required IAM roles and permissions but not able to pull the Cloud Watch Logs into on-premises Splunk server.

 

Splunk1.JPGSplunk2.JPG

Tried same by installing the Splunk on AWS EC2 and Assign the role to EC2 instance and working fine.
Can you please help on this? I have searched on the internet regarding the same but not not the concrete solution for this.

I will appreciate your help.

Thank You
Suraj Shinde

0 Karma

vulnfree
Explorer

Hi pappjr - does your on-prem Splunk server need to be in the DMZ for this to work?

0 Karma

jpapp
New Member

Hi @vulnfree if you're using the Splunk Add-on for AWS then you DO NOT need to be in a DMZ. Your server just needs outbound internet access to make the API calls.

There are other methods to collect this data (most notably Splunk AWS Project Trumpet) that would require a server in the DMZ for collection. I would recommend using Splunk AWS Project Trumpet if you are expecting very high (>100 GB/day) data volumes. Otherwise you will probably find the Splunk Add-on for AWS to be simpler to configure.

0 Karma

vulnfree
Explorer

Hi @jpapp! Thank you! Do you know if this will have AWS charges?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...