Building for the Splunk Platform

Min IOPS Requirement for 1 TB indexing /per day

Prakash493
Communicator

Hi , Where can i find the Min IOPS requirement for the data ingestion of 1 TB/DAY with 12 indexers ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Prakash493,
Splunk recommends at least 800 IOPS, but all you can have it's obviously better!
for more infos see at https://docs.splunk.com/Documentation/Splunk/8.0.0/Capacity/Referencehardware

Ciao.
Giuseppe

0 Karma

Prakash493
Communicator

Yes but it doesn't say about the data ingestion per day. It might be 1 GB/DAY or 100 GB/DAY , So i am looking to best iops for 1 TB/DAY ingestion requirement.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Prakash493,
as I said Splunk requires at least 800 IOPS, but if you have to index 1TB/day: storage is usually the bottleneck but to index much data you have to use many CPUs and rAM.
Splunk requires:

  • as minimun specifications 12 CPUs and 12 GB RAM,
  • as mid range specifications 24 CPUs and 64 GB RAM,
  • as high performance specifications 48 CPUs and 128 GB RAM.

the question is: how many Indexers you have to use?
Remembering that Indexers work also to answer to searches (users and scheduled).

So to index 600-1000 GB/day you should use:

  • 3 Indexers, if you have up to 8 users,
  • 4 Indexers, if you have up to 16 users,
  • 6 Indexers, if you have up to 24 users,
  • 7 Indexers, if you have up to 48 users,

If you have more than 1TB/day, capacity planning grows, 1-2 TB/day:

  • 7 Indexers, if you have up to 4 users,
  • 8 Indexers, if you have up to 8 users,
  • 10 Indexers, if you have up to 16 users,
  • 12 Indexers, if you have up to 24 users,
  • 14 Indexers, if you have up to 48 users,

Then is also relevant how many scheduled searches you have running, or if you have many apps like Enterprise Security or ITSI that contain many scheduled searches requires additional resources.
A complete training about capacity planning is one of the themes of Splunk Architect Training (that I hint!).

Anyway returning to your question: Splunk requires at least 800 IOPS or more (obviously better!), that means 8x15K rpm SAS drives in RAID 1+0 configuration or SSD disks.
Then analyze your requirements and plan the correct number and resources of your Indexers (and Search Heads).

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Streamline Data Ingestion With Deployment Server Essentials

REGISTER NOW!Every day the list of sources Admins are responsible for gets bigger and bigger, often making the ...

Remediate Threats Faster and Simplify Investigations With Splunk Enterprise Security ...

REGISTER NOW!Join us for a Tech Talk around our latest release of Splunk Enterprise Security 7.2! We’ll walk ...

Introduction to Splunk AI

WATCH NOWHow are you using AI in Splunk? Whether you see AI as a threat or opportunity, AI is here to stay. ...