Getting Data In

How to send AWS and non-AWS logs from S3 bucket to Splunk

adnankhan5133
Communicator

Hello,

We have a variety of different AWS logs (i.e. CloudWatch, Cloudtrail, Config, VPC Flow, Aurora) and non-AWS logs (i.e. Palo Alto, Trend Micro) routed to S3 buckets today. There are a total of 15 S3 buckets (5 per AWS account).

Upon recently purchasing and configuring an on-premise Splunk ES (distributed deployment w/ index clustering, no SH clustering yet), our goal is to begin forwarding these logs to our Splunk deployment.

What are some considerations that we should keep in mind? Since we're going with a push approach, we're planning to do the following - Could someone confirm if this looks right? I'm open to suggestions.

  1. Send the logs from the S3 bucket to Amazon Kinesis Firehouse

  2. Firehose writes a batch of events to Splunk via HEC.

  3. Since indexers are not in an AWS VPC (they reside in a separate Oracle Cloud instance), I'm assuming that an SSL certificate needs to be installed on each indexer? We have 1 index cluster in a Production environment and a separate one in our Disaster Recovery environment.

  4. Assign DNS name that resolves to the set of indexers which shall collect data from Kinesis Firehose.

  5. Install Splunk Add-on for Amazon Kinesis on Enterprise and ES Search Head, as well as Cluster Master

  6. Ensure new index is created for AWS logs (1 sourcetype for each AWS log source) and existing indexes are used for the Palo Alto and Trend Micro logs. If new indexes are needed for Palo Alto and Trend Micro logs, I'm assuming that they would still adhere to the appropriate Splunk ES data models.

  7. Configure HEC and create new HEC Token. There will be a unique HEC token per sourcetype.

  8. Configure Amazon Kinesis Firehouse to send data to Splunk.

  9. Ensure all events are backed up to S3 bucket until it is confirmed that all events are processed by Splunk

  10. Search for data by source type to confirm that it is being indexed and visible.

Labels (1)
Tags (2)
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...