Best Practice of Deploying Splunk on Large Infrastructure

[dantimola]

Hi All,

Good Day, currently our Splunk Infrastructure is built with 3 Heavy Forwarders, 6 Non-clustered Indexers, and 2 Clustered Search Heads, our data sources is huge, our Splunk is currently handling almost 8k+ of log sources (servers, network elements etc.) everyday and is still growing, we almost spent 500 GBs of data every day, and our current license is 600 GB. We are planning to do a migration this year and this is our proposed infrastructure.

• 2 Deployment Server
• 12 Heavy Forwarders (per site e.g, Log sources from NewYork site = 2 HFs, Log sources from Canada site = 2 HFs....)
• 6 Clustered Indexers
• 2 Clustered Search Heads

I just want to seek recommendations, suggestions etc. Are we doing it right? Thank you for the answers in advance.

Cheers,
Dan

[sloshburch]

There's a lot of items in this proposal that are of concern. We recently starting introducing Splunk Validated Architectures to guide you in this effort. I also recommend working with your account team who can provide a more intimate conversation about what's going to be most successful for your deployment given the quirks of your data flow and future possibilities.

[bcyates · Accepted solution]

You will need at least 3 SH's in a search head cluster. This is because your search heads have to have a majority vote to elect a captain amongst themselves. You cannot have a majority with 2 SH's.

In most cases, a universal forwarder is preferred over a heavy forwarder. Heavy forwarders drastically increase network IO as compared to a UF. You really only need to use a HF for heavy weight add-ons like DBconnect, or sending logs to a third party source as a UF cannot do either of these things.

But as previously mentioned, you should consider reaching out to Splunk PS to help with your migration.

[skoelpin]

Any update on this?

[jkat54]

For these situations I recommend hiring professional services through Splunk or a partner. It’s not a solid idea to crowd source these details because there are too many variables to be considered.

Another option is to seek your own architecture certification via the splunk education courses. However, it’s better to already have the experience or hire someone who does in my humble opinion. Cheers and best of luck!

[skoelpin]

Whats the purpose of having 12 HF's? Are you wanting a lot of heavy forwarders so you can send cooked data over WAN to the indexers?

You should add an additional SH member as you will run into issues electing a captain with 2 members since its needs majority. Also, where is your deployer, master node, and MC? Perhaps you could use 1 deployment server and use the other as a deployer/master node

[adonio]

hello there,
i only see a change in the number of HF, any particular reason to use 12 HF?
also, 2 clustered SH seems odd, minimum best practice for Search Head Cluster is 3
where is your license master? deployer? (if SH are clustered) Cluster Master? (for indexers)
do you need / want an MC? (Monitoring Console)
do you plan to increase license usage / data ingestion?
are you using a premium app? ES? ITSI?
what are your HA / DR requirements?
please share some more so we can assist