What architecture will work in this Splunk Distributed Environment ?

[sivaksk147]

Hi Team,

I have an infrastructure located globally multiple sites around 10 to 15 Sites which can be generated approximately 1 TB of log volume a day, I would need Splunk expertise suggestions on what architecture will suite for this use case, I have given below few options it would great someone give me inputs on this.

Options 1

1. Setup Heavy Forwarders on each location with Load balance
2. Setup of Indexer cluster and search head cluster at Main Datacenter

WAN Link speed 20-30 Mbps from each site

all location of Heavy Forwarders will get the data from individual local site devices and sent to main data center Index cluster peers nodes and Search head will configure to perform all search events and data visualizations by pulling data from main data center indexer cluster.

Option 2

1. Setup Heavy Forwarders on each locations with Load balance
2. Setup Indexer Cluster on each location
3. Setup search head cluster at main data center

WAN Link speed 20-30 Mbps from each site

All location of heavy forwarders will get the data from individual local site devices and sent to individual data sites index cluster peer nodes and search head cluster at main data center configure to pull data from all the location index cluster and perform search operations and data visualization

Option 3

1. Setup Heavy Forwarders on each locations with Load balance
2. Setup Indexer Cluster at each location
3. Setup a single search head at each location
4. Setup Search head cluster at main data center

WAN Link speed 20-30 Mbps from each site

All location of heavy forwarders will get the data from individual local site devices and sent to individual data sites index cluster peer nodes and local search heads are configured to search events from there individual local sites, and Main data center search head cluster configure to have centralized dashboard from all search head data.

[DalJeanis · Accepted solution]

This is a big wide question, and any competent contractor will tell you the answer is "It depends". Also, you are missing other potential options.

The question of "where do I set up my indexers" will depend heavily on something you didn't tell us... Who will be searching the data, how much, and from where? Even if you knew the answer for 6-12 months out, you probably aren't going to know the answer for 2 years out, so your design has to accommodate both growth and organizational changes.

Other potential configurations include - if each data center will be searching its own data heavily and never searching the other data centers, and the home office will be searching them all, then you could have a local set of HFs that send the data to a local cluster, and also to the home cluster. This costs more license, of course, but it might give much superior performance compared to trying to give all data centers access to all data.

Likewise, it would be possible to have one multisite cluster per data center, with one sub-cluster at the data center and the other sub-cluster at the main data center. A search head cluster at the main data center could be set up to search all of the subclusters located there.

In any case, a multisite clustered implementation like this is something that should not be attempted by someone who has to ask us which way to go. You really need to pull in a professional Splunk architect who can give you (at least) the right words of warning to tell management about the best options. Because there are always words of warning with multisite clustering.

And, at the very least, get yourself onto the Splunk Slack channel, get down to the #clustering subchannel, and ask there.

[sivaksk147]

Thanks for your support.

[DalJeanis]

Sure. There's also an #architect subchannel. You can post this kind of questions in either one - #architect or #clustering, and let the other subchannel know the question is there.

[adonio]

hello there,
when you say "use case" you barely mentioning you have 10-15 sites and ~1TB per day data
what is your use case? do you need HA or DR solution? if so, do you need it for your search tier? indexer tier? all tiers? how many users are you anticipating? where are these users logged in from? are you allowed to transfer data between location? are some data centers restricted (GDPR)? who needs to see what? (search perspective)
i think that your question is very (very) broad and requires a deeper detailed discussion.
with that being said, if you are purchasing (or already purchased) 1TB of splunk license, i am positive you have full access to a Splunk SE who can go over all your requirements and assist you with designing an architecture that will address your "use case"

hope it helps

[sivaksk147]

Hi Adonio,

Thank you very much for your response, i have not procured the licenses yet i am just exploring the possible options at this movement

it would be great if you can able to suggest with below given details.

what is your use case? : I am planning to use Splunk for SIEM only
do you need HA or DR solution? I : would be looking for HA for Indexer tier and Search tier
how many users are you anticipating?:: Approximate users are 40
where are these users logged in from?: Everyone would login from main data center
are you allowed to transfer data between location?: few of the location can be allowed and few of them not allowed
are some data centers restricted (GDPR)?:Few data centers are restricted with GDPR

[adonio]

i really recommend to have a very detailed conversation with a Splunk SE.
many variables here to consider before making any recommendations, especially when you mention some data is under security / GDPR policies. this means you can not have all data in one spot, and also means you can not search all data from one location.
again i will highly recommend to contact either your Splunk SE or a Splunk Partner SE / SME /Consultant and have a serious and detailed discussion.

[sivaksk147]

Thanks for your support