Splunk Service Getting Down Suddenly

medsy — Tue, 02 Mar 2021 10:32:35 GMT

Hi,

So I have an issue with my Splunk Enterprise deployment. I have three instances on my architecture, a Search Head, an Indexer and another Search Head dedicated for Splunk Enterprise Security.

The issue is The service of splunk (splunkd) is getting down suddenly. There is no error in the deployments.

If someone have any explanation or suggestion I'm open to hear it.

Re: Splunk Service Getting Down Suddenly

richgalloway — Tue, 02 Mar 2021 13:46:10 GMT

Splunkd.log should have a log message explaining the sudden disappearance. If it does not then check /var/log/messages for OOM (Out Of Memory) Killer messages.

Re: Splunk Service Getting Down Suddenly

medsy — Tue, 02 Mar 2021 15:21:23 GMT

I got those message on splunk log :

JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a764635f76e33232_at_1614165600_20923
02-24-2021 13:43:08.726 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614132000_11964
02-24-2021 13:43:08.727 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614049200_17022
02-24-2021 13:43:08.815 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614135600_12923
02-24-2021 13:43:08.829 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614045600_15966
02-24-2021 13:43:09.068 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a764635f76e33232_at_1614165600_20923
02-24-2021 13:43:09.081 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614132000_11964
02-24-2021 13:43:09.082 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614049200_17022
02-24-2021 13:43:09.117 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614135600_12923
02-24-2021 13:43:09.124 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614045600_15966
02-24-2021 13:43:36.734 +0100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 10468 - data_source="/opt/splunk/var/log/splunk/audit.log", data_host="svlsplunkses", data_sourcetype="splunk_audit"
02-24-2021 13:44:06.793 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:44:06.897 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:44:08.345 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:03.609 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:03.681 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:05.010 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:05.087 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:06.970 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:07.041 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.149 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.152 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:07.256 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.697 +0100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 10955 - data_source="/opt/splunk/var/log/splunk/audit.log", data_host="svlsplunkses", data_sourcetype="splunk_audit"
02-24-2021 13:45:08.477 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:08.547 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:09.609 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:09.672 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.

Re: Splunk Service Getting Down Suddenly

data_beast — Wed, 22 Sep 2021 10:18:26 GMT

Anyone knows something about below errors ?

WARN JobsFeed - Custom progress indicator signaled progress of > 100%

Unfortunately not explained in docs 😞

Re: Splunk Service Getting Down Suddenly

richgalloway — Wed, 22 Sep 2021 12:17:34 GMT

Please don't hijack threads. Post a new question.

Re: Splunk Service Getting Down Suddenly

data_beast — Wed, 22 Sep 2021 12:27:16 GMT

Is it a hijacking ? I've mentioned error msg pointed out by @medsy above.

@richgalloway - do you know anything about them ?

Re: Splunk Service Getting Down Suddenly

richgalloway — Wed, 22 Sep 2021 14:39:51 GMT

Yes, it's a hijacking. The OP is about Splunk going down, not about a specific log message.

topic Re: Splunk Service Getting Down Suddenly in Splunk Search

Splunk Service Getting Down Suddenly

Re: Splunk Service Getting Down Suddenly

Re: Splunk Service Getting Down Suddenly

Re: Splunk Service Getting Down Suddenly

Re: Splunk Service Getting Down Suddenly

Re: Splunk Service Getting Down Suddenly

Re: Splunk Service Getting Down Suddenly