Configuration initialization for Path took longer than expected|warning

[jat_ashish]

WARN [Indexer] Configuration initialization for C:\Program Files\Splunk\var\run\searchpeers\Seachheadbundle took longer than expected (1359ms) when dispatching a search with search ID remote_searchhead_user__usert_bundle. This usually indicates problems with underlying storage performance.

hi all ,

we have a 6(windows,RAID storage) indexers , 1CM(windows),1DS(windows) and 2(windows) search head.

we are getting above warning every time when we run any search . The warning is from every indexers.

please help us in resolving above warning.

@kamlesh_vaghela @splunk @BSplunk @gcusello @to4kawa @renjith_nair @ITWhisperer 

[gcusello]

Hi @jat_ashish,

as @richgalloway said the first thing is to understand what's the throughtput of your storage: Splunk requires at least 800 IOPS (better 1200).

You can measure it using a tool as Bonnie++ .

Anyway:

• which RAID are you using? Splunk hints RAID0 or RAID1 or RAID10, never RAID5!
• why are you using Windows Server as operative system? Splunk is compatible and certified on Windows Server, but it's useful for test , it isn't a good idea to use it in production systems!

Then, as @richgalloway said, you should see, using the Splunk Monitor Console, how much your Indexers are overloaded and which queues you have on Indexers.

Ciao.

Giuseppe

[jat_ashish]

@gcusello @richgalloway 

1) IOPS is 2000

2) RAID 10

[gcusello]

Hi @jat_ashish,

are you sure of 2000 IOPS?

how do you measured it?

did you used Bonnie++?

which kind of disks are you using? 

I'm asking these things because 2000 IOPS are very very many, for your comprehension: a SATA disk with 15k rpm has around 100 IOPS.

You can monitor performaces also using the Monitor Console app.

As i said Windows isn't a good operative system for a (Splunk) Server!

Anyway the message says that's a storage problem, but, only for completeness, how many CPUs and RAM have you on Indexers and Search Heads?.

Ciao.

Giuseppe

[jat_ashish]

@gcusello yes I asked my IT team.

SH 64gb and 24pysical/48virtual cpu cores

indexers 64gb 24pysical/48virtual cpu cores

[gcusello]

Hi @jat_ashish,

this confirms that it shouldn't be a resources problem.

It remains the storage performance, please check the IOPS as me an @richgalloway said.

Ciao.

Giuseppe

[richgalloway]

Tell us more, please.  What is the disk I/O rate on the search heads and indexers?  How busy are the indexers?  How big is the search bundle?