Splunk Search

Splunk Service Getting Down Suddenly

medsy
Explorer

Hi,

So I have an issue with my Splunk Enterprise deployment. I have three instances on my architecture, a Search Head, an Indexer and another Search Head dedicated for Splunk Enterprise Security.

The issue is The service of splunk (splunkd) is getting down suddenly. There is no error in the deployments.

If someone have any explanation or suggestion I'm open to hear it. 

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunkd.log should have a log message explaining the sudden disappearance.  If it does not then check /var/log/messages for OOM (Out Of Memory) Killer messages.

---
If this reply helps you, Karma would be appreciated.

medsy
Explorer

I got those message on splunk log :

JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a764635f76e33232_at_1614165600_20923
02-24-2021 13:43:08.726 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614132000_11964
02-24-2021 13:43:08.727 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614049200_17022
02-24-2021 13:43:08.815 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614135600_12923
02-24-2021 13:43:08.829 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614045600_15966
02-24-2021 13:43:09.068 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a764635f76e33232_at_1614165600_20923
02-24-2021 13:43:09.081 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614132000_11964
02-24-2021 13:43:09.082 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614049200_17022
02-24-2021 13:43:09.117 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD542c307ea0744c18c_at_1614135600_12923
02-24-2021 13:43:09.124 +0100 WARN JobsFeed - Custom progress indicator signaled progress of > 100% for sid=scheduler__admin_U0EtRW5kcG9pbnRQcm90ZWN0aW9u__RMD5a6fe1e3b4418dcd2_at_1614045600_15966
02-24-2021 13:43:36.734 +0100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 10468 - data_source="/opt/splunk/var/log/splunk/audit.log", data_host="svlsplunkses", data_sourcetype="splunk_audit"
02-24-2021 13:44:06.793 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:44:06.897 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:44:08.345 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:03.609 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:03.681 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:05.010 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:05.087 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:06.970 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:07.041 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.149 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.152 +0100 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
02-24-2021 13:45:07.256 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:07.697 +0100 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 10955 - data_source="/opt/splunk/var/log/splunk/audit.log", data_host="svlsplunkses", data_sourcetype="splunk_audit"
02-24-2021 13:45:08.477 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:08.547 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:09.609 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.
02-24-2021 13:45:09.672 +0100 WARN DispatchManager - The instance is approaching the maximum number of historical searches that can be run concurrently.

 

 

data_beast
Explorer

Anyone knows something about below errors ?

WARN JobsFeed - Custom progress indicator signaled progress of > 100%

Unfortunately not explained in docs 😞

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please don't hijack threads.  Post a new question.

---
If this reply helps you, Karma would be appreciated.
0 Karma

data_beast
Explorer

Is it a hijacking ? I've mentioned error msg pointed out by @medsy above.

@richgalloway - do you know anything about them ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, it's a hijacking.  The OP is about Splunk going down, not about a specific log message.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...