topic Min IOPS Requirement for 1 TB indexing /per day in Splunk Dev

Min IOPS Requirement for 1 TB indexing /per day

Prakash493 — Fri, 06 Dec 2019 16:40:39 GMT

Hi , Where can i find the Min IOPS requirement for the data ingestion of 1 TB/DAY with 12 indexers ?

Re: Min IOPS Requirement for 1 TB indexing /per day

gcusello — Fri, 06 Dec 2019 16:47:01 GMT

Hi @Prakash493,
Splunk recommends at least 800 IOPS, but all you can have it's obviously better!
for more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.0/Capacity/Referencehardware

Ciao.
Giuseppe

Re: Min IOPS Requirement for 1 TB indexing /per day

Prakash493 — Fri, 06 Dec 2019 16:49:26 GMT

Yes but it doesn't say about the data ingestion per day. It might be 1 GB/DAY or 100 GB/DAY , So i am looking to best iops for 1 TB/DAY ingestion requirement.

Re: Min IOPS Requirement for 1 TB indexing /per day

gcusello — Sat, 07 Dec 2019 10:26:38 GMT

Hi @Prakash493,
as I said Splunk requires at least 800 IOPS, but if you have to index 1TB/day: storage is usually the bottleneck but to index much data you have to use many CPUs and rAM.
Splunk requires:

as minimun specifications 12 CPUs and 12 GB RAM,
as mid range specifications 24 CPUs and 64 GB RAM,
as high performance specifications 48 CPUs and 128 GB RAM.

the question is: how many Indexers you have to use?
Remembering that Indexers work also to answer to searches (users and scheduled).

So to index 600-1000 GB/day you should use:

3 Indexers, if you have up to 8 users,
4 Indexers, if you have up to 16 users,
6 Indexers, if you have up to 24 users,
7 Indexers, if you have up to 48 users,

If you have more than 1TB/day, capacity planning grows, 1-2 TB/day:

7 Indexers, if you have up to 4 users,
8 Indexers, if you have up to 8 users,
10 Indexers, if you have up to 16 users,
12 Indexers, if you have up to 24 users,
14 Indexers, if you have up to 48 users,

Then is also relevant how many scheduled searches you have running, or if you have many apps like Enterprise Security or ITSI that contain many scheduled searches requires additional resources.
A complete training about capacity planning is one of the themes of Splunk Architect Training (that I hint!).

Anyway returning to your question: Splunk requires at least 800 IOPS or more (obviously better!), that means 8x15K rpm SAS drives in RAID 1+0 configuration or SSD disks.
Then analyze your requirements and plan the correct number and resources of your Indexers (and Search Heads).

Ciao.
Giuseppe

Re: Min IOPS Requirement for 1 TB indexing /per day

DataOrg — Sat, 22 Feb 2025 17:06:01 GMT

@gcusello may I know how you derive the calculation on the number of indexer for more than 1 TB volume?

Re: Min IOPS Requirement for 1 TB indexing /per day

gcusello — Sun, 23 Feb 2025 06:25:52 GMT

Hi @DataOrg ,

I used some tables from the Architect Training course, but, as I said, it depends on the number of scheduled searches, from the number of concurrent users and from the presence of Premium apps.

Ciao.

Giuseppe