topic Re: Error in search: Configuration initialization for /opt/splunk/etc in Splunk Search

Error in search: Configuration initialization for /opt/splunk/etc

Said7 — Fri, 09 Mar 2018 23:09:45 GMT

Hi,

I have an issue with about a searching, someone know about it, this is the issue:

Error in search: "Configuration initialization for /opt/splunk/etc took longer than expected (XXX ms) when dispatching a search (ID) this typically reflects underlying storage performance issues"

Re: Error in search: Configuration initialization for /opt/splunk/etc

strive — Sat, 10 Mar 2018 00:49:36 GMT

This typically happens when search processes take time to read initial configuration information from disk. Did you check the utilization numbers on your server? Do you have sufficient system resources available? Did splunk start cleanly without any warnings or errors?

Re: Error in search: Configuration initialization for /opt/splunk/etc

deepashri_123 — Tue, 29 Sep 2020 18:27:57 GMT

Hey Said7,

This message means your search processes are taking much time to read initial configuration information from disk. What does the I/O subsystem underneath $SPLUNK_HOME/etc look like in your environment? If $SPLUNK_HOME/etc is networked storage, for example, there might be disk/network performance issues affecting search startup time.

Let me know if this helps!!

Re: Error in search: Configuration initialization for /opt/splunk/etc

mayurr98 — Sat, 10 Mar 2018 05:43:58 GMT

When you create a search, it creates a "search bundle" that contains all the knowledge objects associated with that search (lookups, etc.).

/opt/splunk/etc should have been populated with a file system path to the bundle.

My guess is you have bad file system permissions and need to recursively chown your splunk directory and insure the proper user is running splunk.
have a look at this accepted answer
let me know if this helps!

Re: Error in search: Configuration initialization for /opt/splunk/etc

snigdha9nov — Fri, 25 Jan 2019 13:34:59 GMT

how to resolve this issue ...facing same problem

Re: Error in search: Configuration initialization for /opt/splunk/etc

rmanrique — Thu, 09 Jul 2020 16:26:44 GMT

But finally how do I solve this problem?

Re: Error in search: Configuration initialization for /opt/splunk/etc

rphillips_splk — Tue, 01 Nov 2022 23:55:10 GMT

This message suggests there may be storage performance issues for the path mentioned on the Search Head. You may want to check the SH disk await times (ms) with the search below for the mount point where $SPLUNK_HOME is mounted. Below 10ms is generally considered good performance. If you are seeing await times much higher you should confirm if the mount is NVME and if not, look to move the SH $SPLUNK_HOME mount to NVME for better performance (this is where bundles, search artifacts etc are stored).

| tstats max(data.avg_total_ms) as avg_total_ms where component::iostats sourcetype=splunk_resource_usage index=_introspection host IN ("*SH1*") by host data.mount_point _time span=60s | timechart span=60s max(avg_total_ms) by data.mount_point | eval ideal_latency=10

Some options for looking further into disk performance issues:

The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.

iostat -t -x 2 1800 >> /tmp/iostats.txt

Iotop is an open source and free utility similar to top command, that provides an easy way to monitor Linux Disk I/O usage details and prints a table of existing I/O utilization by process or threads on the systems.

iotop -botqqq --iter=1800 >> /tmp/iotop.log

Re: Error in search: Configuration initialization for /opt/splunk/etc

rphillips_splk — Tue, 01 Nov 2022 23:59:59 GMT

you will need to install the following packages for the above commands

yum install iotop -y
yum install sysstat -y