Search Head cluster : Multi-Cloud Search

[koshyk]

I've been out of touch with Core Splunk for sometime, so just checking if there are options for below requirement

Organisation is looking for RFP for various Big Data products and Organisation needs

-  multi-cloud design for various applications. Application (and thus data) resides in AWS/Azure/GCP in multiple regions within Europe

- Doesn't want to have lot of egress cost. So aggregating data into the cloud which Splunk was installed predominently is out of question.

- The design is to have 'Data nodes' (Indexer clusters or Data clusters) in each of the application/data residing cloud providers

- A Search Head cluster (Cross Cloud search) will be then spun in the main provider (eg AWS), which can then search ALL these remote 'Data nodes'

Is this design feasible in Splunk? (I understand Mothership add-on, but my last encouter with it at enterprise scale was not that great)

Looking for something like below with low latency

[PickleRick]

At first glance it seems like a use case for federated search. Having said that - I've never used federated search myself and can't tell you what limitations it has.

[koshyk]

Agree Guiseppe. I was thinking if anyone in the community have done similar

>> if yes you have to create a multisite cluster, one for each different Cloud, configuring replications between them and configuring Search Heads, in each Cloud, to search in all the clusters.

Creating SHC and replications is not the problem; but the key is for the end-user, they have to Search from a single Search Head cluster and it should search in ALL the clusters in multi-cloud. (This could be SHC on top of another SHCs in each cloud or preferably directly to indexers on each clouds if feasible)

Org. doesn't need data to be replicated between clouds, but we can create clusters within each cloud provider for Replicator factor.

[gcusello]

Hi @koshyk ,

ok,

as I said, you have to create a cluster in each environment.

Then Each Search Head Cluster, muste be configured as Search Head of each Indexer Cluster.

For more details see at https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Configuremulti-clustersearch

let me know if I can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[koshyk]

so what you are saying,  just configure 'indexer clusters' in Each cloud environment and then use a 'SHC' from any of the cloud to search the 'indexer clusters' in ALL cloud environments? You sure it won't causes latency at time of SH aggregation?

A diagram would be really appreciated

[PickleRick]

Depends on what you mean by latency. If it's a pure network-level latency you mean then it's up to you to verify what latency you have between those environment. And no architecting can overcome that.

But of course in terms of egress data, if you just set many different environments in different clouds as peers for a single SH(C), you'll get a lot of traffic since each time your search hits a centralized command it will have to send all results it has so far to the SH layer.

[gcusello]

Hi @koshyk ,

at first, this isn't a question for the Community, but for a Certified Splunk Architect or a Splunk Professional Services Specialist.

Anyway, yes it's possible, but you should define one question: do you want to replicate data across the clouds or not?

if yes you have to create a multisite cluster, one for each different Cloud, configuring replications between them and configuring Search Heads, in each Cloud, to search in all the clusters.

If not, you have to create a cluster in each Cloud and only configure Search Heads, in each Cloud, to search in all the clusters.

Anyway, ths project requires the engagement of a specialist to analyze requirements and design the solution.

Ciao.

Giuseppe