Can we downsize our indexer cluster?

[LovingSplunk]

We are at 10 indexers ingesting around 400 GBs/day. A homogeneous environment with 1 millisecond wait time (I/O Write). How many indexers is it safe to cut?

[PickleRick]

There is no single answer for such question. It mostly depends on your _search_ load. Your ingest volume could be even handled with a single indexer but the search performance would be terrible. So it's mostly about how you're using your Splunk stack. (and obviously how much flexibiity you want/need regarding fault resistance.

[LovingSplunk]

Hi @PickleRick, we are at about 80k searches/day.

[PickleRick]

Again - it depends on the overall load. I have around 6,5k searches on my lab Splunk instance which is very rarely and very lightly used so the number itself is no good indicator. Besides the number of searches it's very important what those searches do, how well (or badly) they are written and so on.

It's different to have 80k searches like "| makeresults | eval _raw="Hello world!"" and completely different to have 80k searches over last two years with heavy join use, badly written search terms, eventstats and so on.

So it's one of those cases where if you don't know on your own, it's best to involve someone with experience and access to your environment (which means either Professional Services or your local Splunk Partner)

[gcusello]

Hi @LovingSplunk ,

it depends on many factors:

• have you ES or ITSI?
• have you a multisite architecture and each site must be able to ingest all the load?
• how many scheduled searches are running on your indexers?

in general you could use one Indexer every 200 GB/day without ES or ITSI and one indexer every 100/150 GB/day with ES or ITSI, but, as I said, it depends on the above factors.

Ciao.

Giuseppe

[LovingSplunk]

Thank you for your response @gcusello. We have ES which is very lightly used in our environment. How can I assess the impact of ES on the environment?

[gcusello]

Hi @LovingSplunk ,

as I said, the hardware specifications require a detailed design activity by a Splunk Architect with experience on ES implementation.

Anyway, at https://help.splunk.com/en/splunk-enterprise-security-8/install/8.4/planning/minimum-specifications-... you can find the minimum hardware requirements for on-premise installations, but they are related to the number of Detections to enable and active user.

Only one very important point of attention: be sure to use a very performant storage, with at least 800 IOPS, otherwise your infrastructure will be your first problem!

Check this point using Bonnie++ or another tool before installing Splunk!

Ciao.

Giuseppe

[isoutamo]

I think that currently (un)official minimum is 1200 IOPS instead of 800. And actually it's better to get as much IOPS as you could get!

[PickleRick]

To be honest I've never been a fan of this IOPS requirement. It's very imprecise. What IOPS are we talking about? At what level? What size of request? So many questions, so few answers.

[LovingSplunk]

Hi @PickleRick, looking at the MC and our IOPS for 10 indexers is barely 300 and we are ingesting 400 GBs/day with those 10 indexers. How many can we cut safely?

[gcusello]

Hi @LovingSplunk .

as @isoutamo said: with 300 IOPS the only way is maintaining your ten IDXs!

Otherwise the performces of your system will be too low to support your system.

Ciao.

Giuseppe

[isoutamo]

If your IOPS is 300 per indexer you cannot cut anything before you get it back to min 800-1200. After that you can probably cut of more than half away.

[gcusello]

Hi @LovingSplunk ,

you can measure IOPS, stopping Splunk on one Indexer and running a tool like Bonnie++, don't use MC.

You must have at least 800 IOPS, better 1200 or more.

Ciao.

Giuseppe