Hardware recommended for indexers

[m_zandinia]

Hi Splunkers!

I currently use 3 indexers in order to ingest my data and respond to search jobs. We use ES in our deployment.
My indexers' hardware is 3 DL38 G7 with 12 physical core and 128GB of RAM. The daily ingested data is 500GB/day although sometimes the ingested data was been over 1.5 TB/day! but this has happened very few times.
I have problems with my ES as the correlation searches always get delayed because of the lack of CPU on my indexers.
Now my company has decided to upgrade the indexers. They suggest me 2 DL560 G10 with 192 physical core and 1.5TB of RAM. That's great! Isn't it?

My only concern is that in my current deployment if one of my indexers goes down I have 2 indexers to ingest data and responding to search jobs but if I replace the old servers with new servers then if one of my indexers goes down I just have one indexer.

So what's your professional recommendation in my circumstances?

[gcusello]

Hi @m_zandinia.

best practices in installing ES say to consider 80 GB/day for each indexer configured in standard way (16 CPUs and 32 GB/RAM).

so if you have 500 GB/day to index, you should have at least 7 indexers.

I think that it isn't the same thing to have 2 Indexers with 48 CPUs, because in this case you'll have disks as bottle neck (Splunk requires at least 800 IOPS, better 1200) and the only way to have high IOPS is to have many disks.

In addition, as you said, a failulre of one Indexer will block your ingestion.

In conclusion, it's better to have 7-8 Indexers with 16/32 CPUs and 32/64 GB/RAM than 2 DL560 G10 with 192 physical core and 1.5TB of RAM.

At least, considering that you have a relevant requirement in terms of log to Index and scheduled searches, I hint to involve a Splunk Architect or Splunk PS to design and valid your architecture, it isn't a job for an answer!

Ciao.

Giuseppe

[m_zandinia]

Thank you for your suggestion. You're right. I do prefer to have more indexers instead of just having 2 huge servers!

[gcusello]

Hi @m_zandinia,

glad to help you, but take in consideration my hint to involve a Splunk Architect or Splunk PS: you haven't a basic requirement, it's a relevant one!

Tell me if I can help more, otherwise, please, accept the answer for the other people of Community.

Ciao and happy splunking.

Giuseppe