Error in search: Configuration initialization for /opt/splunk/etc

[Said7]

Hi,

I have an issue with about a searching, someone know about it, this is the issue:

Error in search: "Configuration initialization for /opt/splunk/etc took longer than expected (XXX ms) when dispatching a search (ID) this typically reflects underlying storage performance issues"

[rphillips_splk]

you will need to install the following packages for the above commands

yum install iotop -y
yum install sysstat -y

[rphillips_splk]

This message suggests there may be storage performance issues for the path mentioned on the Search Head. You may want to check the SH disk await times (ms) with the search below for the mount point where $SPLUNK_HOME  is mounted. Below 10ms is generally considered good performance. If you are seeing await times much higher you should confirm if the mount is NVME and if not, look to move the SH $SPLUNK_HOME mount to NVME for better performance (this is where bundles, search artifacts etc are stored).

| tstats
    max(data.avg_total_ms) as avg_total_ms
   where component::iostats sourcetype=splunk_resource_usage index=_introspection host IN ("*SH1*")
    by host
    data.mount_point
    _time
    span=60s |  timechart span=60s max(avg_total_ms) by data.mount_point | eval ideal_latency=10

Some options for looking further into disk performance issues:

The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.

iostat -t -x 2 1800 >> /tmp/iostats.txt

Iotop is an open source and free utility similar to top command, that provides an easy way to monitor Linux Disk I/O usage details and prints a table of existing I/O utilization by process or threads on the systems.

iotop -botqqq --iter=1800 >> /tmp/iotop.log

[snigdha9nov]

how to resolve this issue ...facing same problem

[mayurr98 · Accepted solution]

When you create a search, it creates a "search bundle" that contains all the knowledge objects associated with that search (lookups, etc.).

/opt/splunk/etc should have been populated with a file system path to the bundle.

My guess is you have bad file system permissions and need to recursively chown your splunk directory and insure the proper user is running splunk.
have a look at this accepted answer
let me know if this helps!

[rmanrique]

But finally how do I solve this problem?

[deepashri_123]

Hey Said7,

This message means your search processes are taking much time to read initial configuration information from disk. What does the I/O subsystem underneath $SPLUNK_HOME/etc look like in your environment? If $SPLUNK_HOME/etc is networked storage, for example, there might be disk/network performance issues affecting search startup time.

Let me know if this helps!!

[strive]

This typically happens when search processes take time to read initial configuration information from disk. Did you check the utilization numbers on your server? Do you have sufficient system resources available? Did splunk start cleanly without any warnings or errors?