Getting Data In

Best Practice of Deploying Splunk on Large Infrastructure

dantimola
Communicator

Hi All,

Good Day, currently our Splunk Infrastructure is built with 3 Heavy Forwarders, 6 Non-clustered Indexers, and 2 Clustered Search Heads, our data sources is huge, our Splunk is currently handling almost 8k+ of log sources (servers, network elements etc.) everyday and is still growing, we almost spent 500 GBs of data every day, and our current license is 600 GB. We are planning to do a migration this year and this is our proposed infrastructure.

  • 2 Deployment Server
  • 12 Heavy Forwarders (per site e.g, Log sources from NewYork site = 2 HFs, Log sources from Canada site = 2 HFs....)
  • 6 Clustered Indexers
  • 2 Clustered Search Heads

I just want to seek recommendations, suggestions etc. Are we doing it right? Thank you for the answers in advance.

Cheers,
Dan

0 Karma
1 Solution

bcyates
Communicator

You will need at least 3 SH's in a search head cluster. This is because your search heads have to have a majority vote to elect a captain amongst themselves. You cannot have a majority with 2 SH's.

In most cases, a universal forwarder is preferred over a heavy forwarder. Heavy forwarders drastically increase network IO as compared to a UF. You really only need to use a HF for heavy weight add-ons like DBconnect, or sending logs to a third party source as a UF cannot do either of these things.

But as previously mentioned, you should consider reaching out to Splunk PS to help with your migration.

View solution in original post

0 Karma

sloshburch
Splunk Employee
Splunk Employee

There's a lot of items in this proposal that are of concern. We recently starting introducing Splunk Validated Architectures to guide you in this effort. I also recommend working with your account team who can provide a more intimate conversation about what's going to be most successful for your deployment given the quirks of your data flow and future possibilities.

0 Karma

bcyates
Communicator

You will need at least 3 SH's in a search head cluster. This is because your search heads have to have a majority vote to elect a captain amongst themselves. You cannot have a majority with 2 SH's.

In most cases, a universal forwarder is preferred over a heavy forwarder. Heavy forwarders drastically increase network IO as compared to a UF. You really only need to use a HF for heavy weight add-ons like DBconnect, or sending logs to a third party source as a UF cannot do either of these things.

But as previously mentioned, you should consider reaching out to Splunk PS to help with your migration.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Any update on this?

0 Karma

jkat54
SplunkTrust
SplunkTrust

For these situations I recommend hiring professional services through Splunk or a partner. It’s not a solid idea to crowd source these details because there are too many variables to be considered.

Another option is to seek your own architecture certification via the splunk education courses. However, it’s better to already have the experience or hire someone who does in my humble opinion. Cheers and best of luck!

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Whats the purpose of having 12 HF's? Are you wanting a lot of heavy forwarders so you can send cooked data over WAN to the indexers?

You should add an additional SH member as you will run into issues electing a captain with 2 members since its needs majority. Also, where is your deployer, master node, and MC? Perhaps you could use 1 deployment server and use the other as a deployer/master node

0 Karma

adonio
Ultra Champion

hello there,
i only see a change in the number of HF, any particular reason to use 12 HF?
also, 2 clustered SH seems odd, minimum best practice for Search Head Cluster is 3
where is your license master? deployer? (if SH are clustered) Cluster Master? (for indexers)
do you need / want an MC? (Monitoring Console)
do you plan to increase license usage / data ingestion?
are you using a premium app? ES? ITSI?
what are your HA / DR requirements?
please share some more so we can assist

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...