WARN [Indexer] Configuration initialization for C:\Program Files\Splunk\var\run\searchpeers\Seachheadbundle took longer than expected (1359ms) when dispatching a search with search ID remote_searchhead_user__usert_bundle. This usually indicates problems with underlying storage performance.
hi all ,
we have a 6(windows,RAID storage) indexers , 1CM(windows),1DS(windows) and 2(windows) search head.
we are getting above warning every time when we run any search . The warning is from every indexers.
please help us in resolving above warning.
@kamlesh_vaghela @splunk @BSplunk @gcusello @to4kawa @renjith_nair @ITWhisperer
Hi @jat_ashish,
as @richgalloway said the first thing is to understand what's the throughtput of your storage: Splunk requires at least 800 IOPS (better 1200).
You can measure it using a tool as Bonnie++ .
Anyway:
Then, as @richgalloway said, you should see, using the Splunk Monitor Console, how much your Indexers are overloaded and which queues you have on Indexers.
Ciao.
Giuseppe
Hi @jat_ashish,
are you sure of 2000 IOPS?
how do you measured it?
did you used Bonnie++?
which kind of disks are you using?
I'm asking these things because 2000 IOPS are very very many, for your comprehension: a SATA disk with 15k rpm has around 100 IOPS.
You can monitor performaces also using the Monitor Console app.
As i said Windows isn't a good operative system for a (Splunk) Server!
Anyway the message says that's a storage problem, but, only for completeness, how many CPUs and RAM have you on Indexers and Search Heads?.
Ciao.
Giuseppe
@gcusello yes I asked my IT team.
SH 64gb and 24pysical/48virtual cpu cores
indexers 64gb 24pysical/48virtual cpu cores
Hi @jat_ashish,
this confirms that it shouldn't be a resources problem.
It remains the storage performance, please check the IOPS as me an @richgalloway said.
Ciao.
Giuseppe
Tell us more, please. What is the disk I/O rate on the search heads and indexers? How busy are the indexers? How big is the search bundle?