Getting Data In

What's the best practice to get AWS data into the Splunk platform at scale?

Splunk Employee
Splunk Employee

What's the best practice to get AWS data, such as VPC Flow, CloudWatch, CloudTrail, into the Splunk platform at scale? The modular inputs in the Splunk Add-on for Amazon Web Services are not sufficient for the scale I need.

0 Karma
1 Solution

Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

The best practice solution is to leverage Splunk AWS Project Trumpet to automate how you collect data for many of these popular data sources in your AWS account.

Splunk AWS Project Trumpet is an open-source tool provided by Splunk that allows you to select the data sources you want to collect, then specify the HEC token where Amazon Kinesis Data Firehose (KDF) should send the events to. Trumpet then deploys a CloudFormation template to create the appropriate AWS resources to begin streaming the events to Splunk HEC.

Although Splunk AWS Project Trumpet leverages Splunk-supported solutions such as the Amazon Kinesis Data Firehose to Splunk integration, it is not a Splunk-supported solution.

You can read more about Splunk AWS Project Trumpet in the blog Automating AWS Data Ingestion into Splunk on Splunk Blogs. You can find the utility itself and additional details at splunk-aws-project-trumpet on Github.

Here are some recommendations for how to implement Splunk AWS Project Trumpet:

  • If you are a Splunk Cloud customer, first file a ticket with Splunk Support to enable HEC for use with Kinesis Firehose. Splunk Cloud operations will provide you with a destination URL to send your KDF data.
  • If you are a Splunk customer running Splunk Enterprise on-prem, we recommend that you deploy a heavy forwarder in your AWS environment with HEC enabled. This enables KDF to forward the data to the heavy forwarder first, then to Splunk Enterprise as a data-collection component.
  • If you are running the Splunk platform in your own AWS environment, deploy a HEC token to all your indexers, and place a classic load balancer in front of the indexers to distribute the KDF traffic across all indexers.

Add-on requirements:

The following video provides a visual walk through of deploying Splunk AWS Project Trumpet in your own environment.
Automating AWS Data Ingestion into Splunk with Project Trumpet

View solution in original post

Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

The best practice solution is to leverage Splunk AWS Project Trumpet to automate how you collect data for many of these popular data sources in your AWS account.

Splunk AWS Project Trumpet is an open-source tool provided by Splunk that allows you to select the data sources you want to collect, then specify the HEC token where Amazon Kinesis Data Firehose (KDF) should send the events to. Trumpet then deploys a CloudFormation template to create the appropriate AWS resources to begin streaming the events to Splunk HEC.

Although Splunk AWS Project Trumpet leverages Splunk-supported solutions such as the Amazon Kinesis Data Firehose to Splunk integration, it is not a Splunk-supported solution.

You can read more about Splunk AWS Project Trumpet in the blog Automating AWS Data Ingestion into Splunk on Splunk Blogs. You can find the utility itself and additional details at splunk-aws-project-trumpet on Github.

Here are some recommendations for how to implement Splunk AWS Project Trumpet:

  • If you are a Splunk Cloud customer, first file a ticket with Splunk Support to enable HEC for use with Kinesis Firehose. Splunk Cloud operations will provide you with a destination URL to send your KDF data.
  • If you are a Splunk customer running Splunk Enterprise on-prem, we recommend that you deploy a heavy forwarder in your AWS environment with HEC enabled. This enables KDF to forward the data to the heavy forwarder first, then to Splunk Enterprise as a data-collection component.
  • If you are running the Splunk platform in your own AWS environment, deploy a HEC token to all your indexers, and place a classic load balancer in front of the indexers to distribute the KDF traffic across all indexers.

Add-on requirements:

The following video provides a visual walk through of deploying Splunk AWS Project Trumpet in your own environment.
Automating AWS Data Ingestion into Splunk with Project Trumpet

View solution in original post

Communicator

Just to clarify; am I reading it correctly that the Splunk Product best practice for getting AWS data into Splunk is not a Splunk-supported solution?

0 Karma

Motivator

No, Splunk created AWS Project Trumpet. The line about Amazon Kinesis isn't supported by Splunk because it's supported by AWS.

License

Cheers,
Jacob
0 Karma