Has anyone scaled Splunk across a multisite deployment? How has it worked out for you and are there capacity planning tips?

[mmensch]

Hello,

I am trying to start a new Splunk project. I have 3 sites across the U.S. I would like to implement Splunk. Each site will have around 16 users. Sometimes these users will be concurrent, but other times it will not (timezones, etc..)

I have done some baseline research, but I would like to know if anyone has scaled Splunk across multiple sites and how it has worked out for them.

I would also like to know how many CPUs & CPU Cores I will need for each search head and Indexer at each site.

Thanks in advance.

[martin_mueller]

The biggest factors in terms of CPU, core, and indexer count are indexing volume and search intensity. Both these factors are impossible to tell from over here, so here's a rough overview from the docs: http://docs.splunk.com/Documentation/Splunk/6.2.3/Capacity/Summaryofperformancerecommendations
Those are for single-site non-redundant Splunk deployments.

Another major factor is redundancy requirements across each site individually, and across all sites combined - should there be one copy overall, one copy per site, several copies per site, will users frequently search data originating from other sites?

Combined with many other factors it's impossible to give you a qualified recommendation through Splunk Answers - consider talking to Splunk Sales to get some technical support and to set up a proof-of-concept for your environment. Sales will also be able to pull in the right engineers to answer these factors with you and come to a sound recommendation.

[martin_mueller]

That's 200-250GB per site and day, right?

For pure volume that'd be two indexers per site, to add mild redundancy you'll want at least three... to allow for 16 truly concurrent users I'd go with four. More if you need high redundancy or intend to use heavy apps such as Enterprise Security. You'll need one Master to manage the indexer cluster overall, not per site.

One very beefy search head per site should be enough to support the users, if you need high availability you'll want three search heads per site in a search head cluster, with a dynamic implicit captain per site amongst those three.

Searching data from other sites will be quick once you configure your multi-site cluster to keep at least one copy of every bucket in each site. Search affinity will make sure that one site's search head will search that site's indexers instead of going over WAN.

As for real-time, in recent versions there have been decent improvements around "indexed real-time", enabled by default... I wouldn't worry about real-time yet.

[mmensch]

Thank you for your input so far Martin. I appreciate it.

Is it OK to implement the Search Head Peers on VMs and what is the correlation between number of CPU cores and VMs and Index Clusters as well?

[martin_mueller]

VMs are okay IFF you actually get at least reference Splunk box specifications for all instances at the same time. That's a very very big if for the real world - Splunk use is quite peaky, when a user launches a couple of searches from a dashboard ALL search peers on that site will jump into action, easily overwhelming the usually oversubscribed hardware underneath your VMs.

You'll want many cores on both indexers and search heads, think 16ish. More than twenty can become hard to fully utilize, scale horizontally instead. Less than twelve may get you bottlenecks.
Take a busy indexer - it'll use four cores for indexing (more soon with pipelinesets) and many cores to fulfil searches. If you only have, say, eight cores then you'll be left with four cores to answer searches which will be painfully slow.

[mmensch]

I've looked over the summary for performance recommendations, however I was curious if there are any multi-site deployments out there that have worked with Splunk's recommendations - in both a development environment and production environment.

I would say each site will be indexing between 2-250GB of data per day.

Each site will also have all users logged in at once running searches. I would prefer the searches to be real-time on the dashboards & real-time alerts. However, I understand that is not always recommended.

Users may be searching items from different sites, so most of the data should be accessible for all users.

[rbal_splunk]

Does these sites need to share data or not?

How is the network connectivity between three sites?

[mmensch]

All of the sites will need to share the data.

I am unsure the exact network connectivity that will be used as of now. My idea is that one master or captain will sit at one of the sites and control all of the sites. Please let me know if this is what should be used or if there is a better option.

[rbal_splunk]

My suggestion wil be for you to use Index clustering --http://docs.splunk.com/Documentation/Splunk/6.2.4/Indexer/Aboutclusters

When you talk about Index Clustering you will need atleast following number of splunk instance

--Cluster Master
-Site 1 : 2 Indexer
-Site 2: 1 or 2 indexer
-Site 3: 1 or 2 indexer
- you need to have Search Head - one in each site.

In your case only conern I wil hae is --- that your user base is quite small 10-15 users per site.
Are you planning to expand that?