having exclamation symbol and showing warning message:indicates problems with underlying storage performance

[Splunk-Star]

We have a Splunk Dashboard for our Team in Splunk  Cluster. Almost every report item is having exclamation symbol and contains the below message. The issue has been present for the past 1 month. Could you please help me in fixing the issue.

Error Details:
---------------------
*-199.corp.apple.com] Configuration initialization for /ngs/app/splunkp/mounted_bundles/peer_8089/*_SHC took longer than expected (1145ms) when dispatching a search with search ID remote_sh-*-13.corp.apple.com_2320431658__232041658__search__RMD578320bc0a7e9dada_1709881516.707_378AAA09-A2C2-4B63-B88A-50A6B29A67DF. This usually indicates problems with underlying storage performance."

[isoutamo]

Hi

as @gcusello said you have performance issue on your splunk system. Quite probably it's on indexer side. Another place could be SH side if you have too small splunk var directory. 

I suppose that you have MC on place? Then use it for monitoring your environment. 

You could look this https://conf.splunk.com/files/2021/slides/TRU1172B.pdf and there are also some other MC and CMC presentations and those contains links to other resources and instructions.

If those didn't help, then ask help from PS or some Splunk architect.

r. Ismo

[gcusello]

Hi @Splunk-Star,

you have to check at first your infrastructure: have you the minimal resources required by Splunk?

if yes, you should analyze your  situation and eventually redesign your infrastructure for the new requirements: e.g. if you have many users or you're using many scheduled searches or you're using too real time searches, you have to use more resources (CPUs).

then you have to analyze your configurations, e.g. some time ago I had this issue on Splunk Cloud, but the solution was to redistribute the schedule of the scheduled searches and the percentage of resources for scheduled searches.

In both cases I hint to engare a Splunk Professional Service or a Splunk Architect: this issue requires a good experience in Splunk infrastructures.

Ciao.

Giuseppe