Deployment Architecture

On-Prem and Cloud Hybrid Architecture

Path Finder

We have an on-prem Splunk deployment with 50TB/day ingest, 22PB storage, long term retention (1 year on most indexes). We use Kafka as primary ingest method. We collect cloud logs in Azure using EventHub and Kinesis in AWS. We then use Kafka MirrorMaker to bring cloud logs into on-prem Kafka.

The volume ratio is currently 95% on-prem vs 5% cloud logs by volume. Inevitably this ratio will change in favor of cloud logs. And the cost to bring cloud logs on-prem will rise. For Azure alone we estimate Microsoft charge $15k/month for 5TB/day data egress. And of course the size of our cloud pipe is an issue. Therefore we are considering an architecture which will avoid backhauling cloud logs. Our stakeholders have made it clear they want a seamless experience - no logging into multiple platforms.

Is your organization in a similar position to this or have you overcome this problem? 




Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...