Min IOPS Requirement for 1 TB indexing /per day

[Prakash493]

Hi , Where can i find the Min IOPS requirement for the data ingestion of 1 TB/DAY with 12 indexers ?

[DataOrg]

@gcusello may I know how you derive the calculation on the number of indexer for more than 1 TB volume?

[gcusello]

Hi @DataOrg ,

I used some tables from the Architect Training course, but, as I said, it depends on the number of scheduled searches, from the number of concurrent users and from the presence of Premium apps.

Ciao.

Giuseppe

[gcusello]

Hi @Prakash493,
Splunk recommends at least 800 IOPS, but all you can have it's obviously better!
for more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.0/Capacity/Referencehardware

Ciao.
Giuseppe

[Prakash493]

Yes but it doesn't say about the data ingestion per day. It might be 1 GB/DAY or 100 GB/DAY , So i am looking to best iops for 1 TB/DAY ingestion requirement.

[gcusello]

Hi @Prakash493,
as I said Splunk requires at least 800 IOPS, but if you have to index 1TB/day: storage is usually the bottleneck but to index much data you have to use many CPUs and rAM.
Splunk requires:

• as minimun specifications 12 CPUs and 12 GB RAM,
• as mid range specifications 24 CPUs and 64 GB RAM,
• as high performance specifications 48 CPUs and 128 GB RAM.

the question is: how many Indexers you have to use?
Remembering that Indexers work also to answer to searches (users and scheduled).

So to index 600-1000 GB/day you should use:

• 3 Indexers, if you have up to 8 users,
• 4 Indexers, if you have up to 16 users,
• 6 Indexers, if you have up to 24 users,
• 7 Indexers, if you have up to 48 users,

If you have more than 1TB/day, capacity planning grows, 1-2 TB/day:

• 7 Indexers, if you have up to 4 users,
• 8 Indexers, if you have up to 8 users,
• 10 Indexers, if you have up to 16 users,
• 12 Indexers, if you have up to 24 users,
• 14 Indexers, if you have up to 48 users,

Then is also relevant how many scheduled searches you have running, or if you have many apps like Enterprise Security or ITSI that contain many scheduled searches requires additional resources.
A complete training about capacity planning is one of the themes of Splunk Architect Training (that I hint!).

Anyway returning to your question: Splunk requires at least 800 IOPS or more (obviously better!), that means 8x15K rpm SAS drives in RAID 1+0 configuration or SSD disks.
Then analyze your requirements and plan the correct number and resources of your Indexers (and Search Heads).

Ciao.
Giuseppe