WARN [Indexer] Configuration initialization for C:\Program Files\Splunk\var\run\searchpeers\Seachheadbundle took longer than expected (1359ms) when dispatching a search with search ID remote_searchhead_user__usert_bundle. This usually indicates problems with underlying storage performance.
hi all ,
we have a 6(windows,RAID storage) indexers , 1CM(windows),1DS(windows) and 2(windows) search head.
we are getting above warning every time when we run any search . The warning is from every indexers.
please help us in resolving above warning.
as @richgalloway said the first thing is to understand what's the throughtput of your storage: Splunk requires at least 800 IOPS (better 1200).
You can measure it using a tool as Bonnie++ .
Then, as @richgalloway said, you should see, using the Splunk Monitor Console, how much your Indexers are overloaded and which queues you have on Indexers.
are you sure of 2000 IOPS?
how do you measured it?
did you used Bonnie++?
which kind of disks are you using?
I'm asking these things because 2000 IOPS are very very many, for your comprehension: a SATA disk with 15k rpm has around 100 IOPS.
You can monitor performaces also using the Monitor Console app.
As i said Windows isn't a good operative system for a (Splunk) Server!
Anyway the message says that's a storage problem, but, only for completeness, how many CPUs and RAM have you on Indexers and Search Heads?.
Tell us more, please. What is the disk I/O rate on the search heads and indexers? How busy are the indexers? How big is the search bundle?