topic Re: Splunk Internal logs Errors in Getting Data In

Can someone help me with the solutions for the below errors on splunk internal logs?

Roy_9 — Wed, 16 Aug 2023 23:10:17 GMT

Hello,

Can someone please help me with the solutions for the below errors on splunk internal logs?

Host Component Error
splunk01-shc SearchScheduler The maximum disk usage quota for this user has been reached.
splunk01-shc DispatchManager The maximum number of concurrent historical searches on this instance has been reached.
splunk01-shc UserManagerPro SAML config is invalid, Reconfigure it.
splunk02 ExecProcessor message from "{}" /bin/sh: {}: Permission denied
splunk02 ExecProcessor message from "{}" File "/opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py", line 29, in should_run
splunk01-shc SearchScheduler Search not executed: The maximum number of concurrent historical searches on this instance has been reached., concurrency_category={}, concurrency_context={}, current_concurrency={} concurrency_limit={}

Re: Splunk Internal logs Errors

gcusello — Wed, 16 Aug 2023 09:22:07 GMT

Hi @Roy_9,

you have many issies in your Splunk, but the main and most evident is that you haven't sufficient resources, in terms of disk space, and this issue probably is causing the others.

Then there's also an issue on SAML, but, my hint is to solve the main issue and then the others.

Ciao.

Giuseppe

Re: Splunk Internal logs Errors

Roy_9 — Wed, 16 Aug 2023 18:22:54 GMT

thanks @gcusello i will take a look at the disk space resources and see if it helps

Re: Splunk Internal logs Errors

gcusello — Thu, 17 Aug 2023 06:49:55 GMT

Hi @Roy_9 ,

let me know if I can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Splunk Internal logs Errors

Roy_9 — Tue, 22 Aug 2023 20:10:42 GMT

Sure @gcusello I am still working on this effort, i will definitely post my observations once I am done.

Re: Splunk Internal logs Errors

isoutamo — Tue, 22 Aug 2023 20:47:33 GMT

another thing where you should look is amount of searches on your SHC. Those logs said that there are running more than this node has capacity. You should check if there are skipped/deferred searches and try to modify those to use less resources. Also check if there are real-time searches and/or alerts and convert those to scheduled searches. Probably you have also some users which have quite many searches and maybe too long TTL time for those? When you fix this, you should get more disk space for that user and maybe no more quota overflow?

r. Ismo