Can someone help me with the solutions for the below errors on splunk internal logs?

[Roy_9]

Hello,

Can someone please help me with the solutions for the below errors on splunk internal logs?

Host                     Component                    Error
splunk01-shc SearchScheduler The maximum disk usage quota for this user has been reached.
splunk01-shc DispatchManager The maximum number of concurrent historical searches on this instance has been reached.
splunk01-shc UserManagerPro SAML config is invalid, Reconfigure it.
splunk02 ExecProcessor message from "{}" /bin/sh: {}: Permission denied
splunk02 ExecProcessor message from "{}" File "/opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py", line 29, in should_run
splunk01-shc SearchScheduler Search not executed: The maximum number of concurrent historical searches on this instance has been reached., concurrency_category={}, concurrency_context={}, current_concurrency={} concurrency_limit={}

[gcusello]

Hi @Roy_9,

you have many issies in your Splunk, but the main and most evident is that you haven't sufficient resources, in terms of disk space, and this issue probably is causing the others.

Then there's also an issue on SAML, but, my hint is to solve the main issue and then the others.

Ciao.

Giuseppe

[Roy_9]

thanks @gcusello i will take a look at the disk space resources and see if it helps

[gcusello]

Hi @Roy_9 ,

let me know if I can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[Roy_9]

Sure @gcusello  I am still working on this effort, i will definitely post my observations once I am done.

[isoutamo]

Hi

another thing where you should look is amount of searches on your SHC. Those logs said that there are running more than this node has capacity. You should check if there are skipped/deferred searches and try to modify those to use less resources. Also check if there are real-time searches and/or alerts and convert those to scheduled searches. Probably you have also some users which have quite many searches and maybe too long TTL time for those? When you fix this, you should get more disk space for that user and maybe no more quota overflow?

r. Ismo