topic Re: Configuration initialization for Path took longer than expected|warning in Splunk Search

Configuration initialization for Path took longer than expected|warning

jat_ashish — Tue, 22 Dec 2020 13:36:18 GMT

WARN [Indexer] Configuration initialization for C:\Program Files\Splunk\var\run\searchpeers\Seachheadbundle took longer than expected (1359ms) when dispatching a search with search ID remote_searchhead_user__usert_bundle. This usually indicates problems with underlying storage performance.

hi all ,

we have a 6(windows,RAID storage) indexers , 1CM(windows),1DS(windows) and 2(windows) search head.

we are getting above warning every time when we run any search . The warning is from every indexers.

please help us in resolving above warning.

@kamlesh_vaghela @splunk @Anonymous @gcusello @to4kawa @renjith_nair @ITWhisperer

Re: Configuration initialization for Path took longer than expected|warning

richgalloway — Tue, 22 Dec 2020 13:39:01 GMT

Tell us more, please. What is the disk I/O rate on the search heads and indexers? How busy are the indexers? How big is the search bundle?

Re: Configuration initialization for Path took longer than expected|warning

gcusello — Tue, 22 Dec 2020 14:13:51 GMT

Hi @jat_ashish,

as @richgalloway said the first thing is to understand what's the throughtput of your storage: Splunk requires at least 800 IOPS (better 1200).

You can measure it using a tool as Bonnie++ .

Anyway:

which RAID are you using? Splunk hints RAID0 or RAID1 or RAID10, never RAID5!
why are you using Windows Server as operative system? Splunk is compatible and certified on Windows Server, but it's useful for test , it isn't a good idea to use it in production systems!

Then, as @richgalloway said, you should see, using the Splunk Monitor Console, how much your Indexers are overloaded and which queues you have on Indexers.

Ciao.

Giuseppe

Re: Configuration initialization for Path took longer than expected|warning

jat_ashish — Thu, 14 Jan 2021 15:37:22 GMT

@gcusello @richgalloway

1) IOPS is 2000

2) RAID 10

Re: Configuration initialization for Path took longer than expected|warning

gcusello — Thu, 14 Jan 2021 16:53:20 GMT

Hi @jat_ashish,

are you sure of 2000 IOPS?

how do you measured it?

did you used Bonnie++?

which kind of disks are you using?

I'm asking these things because 2000 IOPS are very very many, for your comprehension: a SATA disk with 15k rpm has around 100 IOPS.

You can monitor performaces also using the Monitor Console app.

As i said Windows isn't a good operative system for a (Splunk) Server!

Anyway the message says that's a storage problem, but, only for completeness, how many CPUs and RAM have you on Indexers and Search Heads?.

Ciao.

Giuseppe

Re: Configuration initialization for Path took longer than expected|warning

jat_ashish — Fri, 15 Jan 2021 05:15:35 GMT

@gcusello yes I asked my IT team.

SH 64gb and 24pysical/48virtual cpu cores

indexers 64gb 24pysical/48virtual cpu cores

Re: Configuration initialization for Path took longer than expected|warning

gcusello — Fri, 15 Jan 2021 07:08:48 GMT

Hi @jat_ashish,

this confirms that it shouldn't be a resources problem.

It remains the storage performance, please check the IOPS as me an @richgalloway said.

Ciao.

Giuseppe