Hello All ,
I have indexer cluster with 5 indexers with different storage space .Indexer 1 has 4.3TB ,Indexer2 has 6.4TB ,Indexer 6.5TB,Indexer 4 and 5 has 88TB .
1) How does the data gets stored in each indexer , does the highest storage indexer takes more data compared to the smaller one
2)Does increasing the index size from default 500GB to 1TB for few indexes causes any issues with data
3)I have the lowest size indexer always complaining about minimum free disk space , does this cause any data loss
4) We have a retention policy of 1year , does the above inconsistent hardware causes loss of data
Thank You in Advance
It's a cluster so the settings are supposed to be the same across the board, that means you have to be configured according to the smallest indexer.
Increasing the index size will not cause issues with the data. However, It will have an effect on your disk storage consumption, your indexers might get full and it will cause your data to be frozen before the intended retention time to free up space. So it is important to understand the Splunk data lifecycle and set things properly like the data retention policy etc. (volume limits too)
The question is , does your smallest indexer can meet your 1 year retention policy ?
Thank You @jimodonald .One last question what would be the default hot bucket setting when an index is created .Below is the index setting we have and what is the setting should we change to make the hot bucket 30 days
coldPath = $SPLUNK_DB/aws_abc/colddb
homePath = $SPLUNK_DB/aws_abc/db
thawedPath = $SPLUNK_DB/aws_abc/thawedb
frozenTimePeriodInSecs = 31536000
The forwarders use an internal load balancing mechanism to determine the target for sending their data. The load balancing mechanism is not configurable.
Ideally, your indexers should all be identical -- same amount of memory; same number of CPUs; same amount of disk capacity for OS, splunk, hot/warm, and cold.
For best planning, based on those storage specifications, you should plan your indexes and retention for the lowest storage amount of 4.3TB, or a total cluster capacity of 21.5 TB (minus the needed storage for your replication factor). You are already seeing the impact of the inconsistent sizing with the errors noted in question 3.
It's not possible to determine if the retention of 1 year is problematic without the details of how your indexes are stored.
I would strongly recommend talking this over with 1) your Splunk Sales Engineer, 2) your local user group experts, or 3) Splunk Professional Services.
In the meantime, please read up on how Splunk clustering works here.