Splunk Service Getting Down Suddenly

[medsy]

Hi,

So I have an issue with my Splunk Enterprise deployment. I have three instances on my architecture, a Search Head, an Indexer and another Search Head dedicated for Splunk Enterprise Security.

The issue is The service of splunk (splunkd) is getting down suddenly. There is no error in the deployments.

If someone have any explanation or suggestion I'm open to hear it. 

[richgalloway]

Splunkd.log should have a log message explaining the sudden disappearance.  If it does not then check /var/log/messages for OOM (Out Of Memory) Killer messages.

[medsy]

I got those message on splunk log :

JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a764635f76e33232_at_1614165600_20923
02-24-2021 13:43:08.726 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614132000_11964
02-24-2021 13:43:08.727 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614049200_17022
02-24-2021 13:43:08.815 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614135600_12923
02-24-2021 13:43:08.829 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614045600_15966
02-24-2021 13:43:09.068 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a764635f76e33232_at_1614165600_20923
02-24-2021 13:43:09.081 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614132000_11964
02-24-2021 13:43:09.082 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614049200_17022
02-24-2021 13:43:09.117 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614135600_12923
02-24-2021 13:43:09.124 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614045600_15966
02-24-2021 13:43:36.734 +0100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 10468 - data_source="/opt/splunk/var/log/splunk/audit.log", data_host="svlsplunkses", data_sourcetype="splunk_audit"
02-24-2021 13:44:06.793 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:44:06.897 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:44:08.345 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:03.609 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:03.681 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:05.010 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:05.087 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:06.970 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:07.041 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.149 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.152 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:07.256 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.697 +0100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 10955 - data_source="/opt/splunk/var/log/splunk/audit.log", data_host="svlsplunkses", data_sourcetype="splunk_audit"
02-24-2021 13:45:08.477 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:08.547 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:09.609 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:09.672 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.

[data_beast]

Anyone knows something about below errors ?

WARN JobsFeed - Custom progress indicator signaled progress of > 100%

Unfortunately not explained in docs 😞

[richgalloway]

Please don't hijack threads.  Post a new question.

[data_beast]

Is it a hijacking ? I've mentioned error msg pointed out by @medsy above.

@richgalloway - do you know anything about them ?

[richgalloway]

Yes, it's a hijacking.  The OP is about Splunk going down, not about a specific log message.