All Apps and Add-ons

Multiple AWS Account integration using Splunk Add ons

shwetas
Explorer

Hi All,

Would like to take a view on Splunk Deployment architecture .

In our environment we have deployed Splunk Enterprise and integrated with AWS,Azure and Google Cloud using Splunk Addons (It’s not a distributed environment)

Would like to know did anyone tried configuring multiple AWS account using Splunk add-on for AWS ,If yes how is the performance and which architecture being used.

Regards,
Shweta

0 Karma
1 Solution

soumyasaha25
Contributor

we do have multiple AWS accounts logging to splunk with considerably good reliability, you just need to get your configuration right.
For AWS a bulk of the data is ingested from CloudWatch log groups using kinesis streams, the differentiating factor in this case is how you decide to get the data into Splunk i.e Pull based or push based approach. the pull based approach (using AWS add 0n) although relatively easy to setup the push based approach with AWS kinesis firehose and splunk HEC is more reliable.

Again, everything depends on your setup, what is the volume of data that you want to onboard, what use cases you want to implement, what AWS services you can implement/use, what is your expectation on data delivery and its reliability, etc.
Once you have these figured out you can decide on which approach you want to use to onboard data optimally.
Refer this to get an overvie on how to onboard Cloudwatch logs
Also, there are a few prdefined Lambda blueprints that you can start out with and customize based on your requirements, see this

Hope this helps. Happy Splunking!!

View solution in original post

0 Karma

shwetas
Explorer

Hi Soumya,
Thanks for the reply, Actually we are also following the same method creating multiple indexes for multiple customer. But I am only concerned about the performance of Splunk.

Ideally as we are collecting data of CSP, So we don't need forwarder at customer premises so can we go with installing and configuring add-ons at Indexer end? How you have done in your environment. If you can just give me brief like where you have configured addons how its being push to search head is it like forwarder->indexer->search head OR indexer->Search head.

Thanks in Advance!!!!!!!!

Shweta

0 Karma

soumyasaha25
Contributor

I don't believe there is a limit on the no of indexes as such. If you worry about the performance it all depends on how your data is coming and how you write your searches. Although, as some of the users have reported you start to run into performance issues if the number of indexes are more than 100K (which quite a huge number and is hardly reached).
regarding your query on where to configure addons/apps - usually, it is mentioned in the app/addon documentation as to where it needs to be installed based on what features/activities are performed by the app/add on.
For example: refer this page for the AWS add on.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Regarding integrating multiple AWS accounts into Splunk, have you seen the recently released Grand Central App? I'd check out this blog post, as it can help you integrate multiple AWS accounts into Splunk : https://www.splunk.com/en_us/blog/partners/trumpeting-to-grand-central-monitor-and-deploy-cloud-base...

0 Karma

shwetas
Explorer

Thanks for the inputs,would also like to know have u used multiple indexes for each account or one index sending data to search head for all the configured account?

Regards,
Shweta

0 Karma

soumyasaha25
Contributor

Yes, we use multiple indexes (one per account per data type).
example: for account xx we have

1. name_xx_vpcflow
2. name_xx_linux 

Also, If your questions are resolved, please make sure you accept the answer to close this question.
Cheers

0 Karma

soumyasaha25
Contributor

we do have multiple AWS accounts logging to splunk with considerably good reliability, you just need to get your configuration right.
For AWS a bulk of the data is ingested from CloudWatch log groups using kinesis streams, the differentiating factor in this case is how you decide to get the data into Splunk i.e Pull based or push based approach. the pull based approach (using AWS add 0n) although relatively easy to setup the push based approach with AWS kinesis firehose and splunk HEC is more reliable.

Again, everything depends on your setup, what is the volume of data that you want to onboard, what use cases you want to implement, what AWS services you can implement/use, what is your expectation on data delivery and its reliability, etc.
Once you have these figured out you can decide on which approach you want to use to onboard data optimally.
Refer this to get an overvie on how to onboard Cloudwatch logs
Also, there are a few prdefined Lambda blueprints that you can start out with and customize based on your requirements, see this

Hope this helps. Happy Splunking!!

0 Karma

akasmika
Loves-to-Learn

@soumyasaha25 , Are you ingesting cloudtrail logs from multiple accounts? If yes, can you please suggest the approach?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...