topic Re: Splunk addon for AWS in All Apps and Add-ons

Splunk addon for AWS

bbiswabhusan — Tue, 26 Mar 2019 08:21:55 GMT

Hello experts, im in the process of onboarding microservices on AWS lambda to splunk. the microservices are logging to cloudwatch at the moment. I have two options currently, using kinesis firehose lambda blueprint and splunk add-on for AWS. Just ant to know what are the specific cases where kinesis firehose must be used as I can see the add-on can process all kins of logs at the moment.
Can someone please guide.
Thanks
Biswa

Re: Splunk addon for AWS

amiracle — Tue, 26 Mar 2019 15:08:26 GMT

We published a white paper to help with this type of question: https://www.splunk.com/en_us/form/getting-data-into-gdi-splunk-from-aws.html .

The TL;DR version of the document for your specific use case : For high volume data, Kinesis Data Firehose (KDF) is one of the better options. If you require validation that the event made it into Splunk and you do not want to setup a HWF, then KDF makes the most sense.

For either lower volume data or data which does not require event validation, but you want to collect it serverlessly, then Lambda Functions make sense.

One last point, is that KDF can also be a less expensive path when compared to using CloudWatch Logs (https://aws.amazon.com/cloudwatch/pricing/ ) vs. (https://aws.amazon.com/kinesis/data-firehose/pricing/ ).

Re: Splunk addon for AWS

bbiswabhusan — Wed, 27 Mar 2019 06:52:15 GMT

Hi @amiracle , thanks for your inputs.all the microservices are logging to cloudwatch at the moment. so we are considering using the splunk add-on for aws as we dont have vpc logs in scope at the moment. Can we go ahead, what is your thoughts on this.
Thanks

Re: Splunk addon for AWS

amiracle — Wed, 27 Mar 2019 13:09:59 GMT

Yes you can collect the CloudWatch Logs this way, but I would use either a Lambda function to collect them or Kinesis streams to avoid hitting your API limits by pulling the data via the Splunk Add-on.