Splunk Search

Error in search: Configuration initialization for /opt/splunk/etc

Said7
Explorer

Hi,

I have an issue with about a searching, someone know about it, this is the issue:

Error in search: "Configuration initialization for /opt/splunk/etc took longer than expected (XXX ms) when dispatching a search (ID) this typically reflects underlying storage performance issues"

1 Solution

mayurr98
Super Champion

When you create a search, it creates a "search bundle" that contains all the knowledge objects associated with that search (lookups, etc.).

/opt/splunk/etc should have been populated with a file system path to the bundle.

My guess is you have bad file system permissions and need to recursively chown your splunk directory and insure the proper user is running splunk.
have a look at this accepted answer
let me know if this helps!

View solution in original post

rphillips_splk
Splunk Employee
Splunk Employee

you will need to install the following packages for the above commands

yum install iotop -y
yum install sysstat -y

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

This message suggests there may be storage performance issues for the path mentioned on the Search Head. You may want to check the SH disk await times (ms) with the search below for the mount point where $SPLUNK_HOME  is mounted. Below 10ms is generally considered good performance. If you are seeing await times much higher you should confirm if the mount is NVME and if not, look to move the SH $SPLUNK_HOME mount to NVME for better performance (this is where bundles, search artifacts etc are stored).

 

| tstats
    max(data.avg_total_ms) as avg_total_ms
   where component::iostats sourcetype=splunk_resource_usage index=_introspection host IN ("*SH1*")
    by host
    data.mount_point
    _time
    span=60s |  timechart span=60s max(avg_total_ms) by data.mount_point | eval ideal_latency=10

 


Some options for looking further into disk performance issues:

The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.

 

 

iostat -t -x 2 1800 >> /tmp/iostats.txt



 

Iotop is an open source and free utility similar to top command, that provides an easy way to monitor Linux Disk I/O usage details and prints a table of existing I/O utilization by process or threads on the systems.

 

 

iotop -botqqq --iter=1800 >> /tmp/iotop.log

 

snigdha9nov
Engager

how to resolve this issue ...facing same problem

0 Karma

mayurr98
Super Champion

When you create a search, it creates a "search bundle" that contains all the knowledge objects associated with that search (lookups, etc.).

/opt/splunk/etc should have been populated with a file system path to the bundle.

My guess is you have bad file system permissions and need to recursively chown your splunk directory and insure the proper user is running splunk.
have a look at this accepted answer
let me know if this helps!

rmanrique
Path Finder

But finally how do I solve this problem?

0 Karma

deepashri_123
Motivator

Hey Said7,

This message means your search processes are taking much time to read initial configuration information from disk. What does the I/O subsystem underneath $SPLUNK_HOME/etc look like in your environment? If $SPLUNK_HOME/etc is networked storage, for example, there might be disk/network performance issues affecting search startup time.

Let me know if this helps!!

0 Karma

strive
Influencer

This typically happens when search processes take time to read initial configuration information from disk. Did you check the utilization numbers on your server? Do you have sufficient system resources available? Did splunk start cleanly without any warnings or errors?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...