Splunk Search

[SUSE] Crash - Splunk SH is not able to perform a search against indexers.

sylim_splunk
Splunk Employee
Splunk Employee

We are testing our upgrade from Splunk 7.3.1 to 8.1.2. Once we upgrade the SH regardless of the indexers' version 7.3.1 or 8.1.2, we are unable to run any searches.

Every time we attempt a search, Splunk crashes 100% - crash logs looks like below;

- Resource usage is low as in CPU IDLE 99% memory usages is less than 10%. 

- systemd and bootstart enabled

 

[build 545206cc9f70] 2021-02-22 17:44:41
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 15454 running under UID 1863478.
Crashing thread: RemoteTimelineReadThread
Registers:
RIP: [0x00007F8BE1E67F77] gsignal + 55 (libc.so.6 + 0x34F77)
RDI: [0x0000000000003C5E]

.. Backtrace (PIC build):
[0x00007F8BE1E67F77] gsignal + 55 (libc.so.6 + 0x34F77)
[0x00007F8BE1E6934A] abort + 314 (libc.so.6 + 0x3634A)
[0x000056191C24136E] ? (splunkd + 0x128F36E)
[0x000056191E355416] _ZN10__cxxabiv111__terminateEPFvvE + 6 (splunkd + 0x33A3416)
[0x000056191E355461] ? (splunkd + 0x33A3461)
[0x000056191E355594] ? (splunkd + 0x33A3594)
[0x000056191C2632C6] ? (splunkd + 0x12B12C6)
[0x000056191C263724] ? (splunkd + 0x12B1724)
[0x000056191D28257B] _ZN6ThreadC1EPKcz + 443 (splunkd + 0x22D057B)
[0x000056191DB44181] _ZN39EventDownloadInitializeCollectionThreadC1EP20EventDownloadManager + 33 (splunkd + 0x2B92181)
[0x000056191DB490A8] _ZN20EventDownloadManagerC2ERK3Str + 600 (splunkd + 0x2B970A8)
[0x000056191D7DF4BF] _ZN9TimelinerC2ERK3Str + 319 (splunkd + 0x282D4BF)
[0x000056191C523782] _ZN24RemoteTimelineReadThread4mainEv + 114 (splunkd + 0x1571782)
[0x000056191D281F87] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22CFF87)
[0x00007F8BE21E1724] ? (libpthread.so.0 + 0x8724)
[0x00007F8BE1F1FEED] clone + 109 (libc.so.6 + 0xECEED)
Linux / ito044165 / 4.4.180-94.130-default / #1 SMP Fri Sep 4 09:11:24 UTC 2020 (e5e5142) / x86_64
C++ exception:
exception_addr=0x7f8be158bfc0
typeinfo=0x56191f0bf6e8, name=15ThreadException

what(): RemoteTimelineReadThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 20 threads active

Labels (1)
Tags (1)
0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

The crash happened 100% for any searches when there's no job running or scheduled at all - pthread_create: Resource temporarily unavailable was not triggered by the memory limit.

In search.log, it also has a message as below;

"02-20-2021 17:44:41.364 WARN Thread - RemoteTimelineReadThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 20 threads active"

 

It was caused by hitting DefaultTasksMax and fixed by adding TasksMax=infinity for the service. Further details found in the link, https://www.suse.com/support/kb/doc/?id=000015901  

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

The crash happened 100% for any searches when there's no job running or scheduled at all - pthread_create: Resource temporarily unavailable was not triggered by the memory limit.

In search.log, it also has a message as below;

"02-20-2021 17:44:41.364 WARN Thread - RemoteTimelineReadThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 20 threads active"

 

It was caused by hitting DefaultTasksMax and fixed by adding TasksMax=infinity for the service. Further details found in the link, https://www.suse.com/support/kb/doc/?id=000015901  

richgalloway
SplunkTrust
SplunkTrust

Does this happen for one particular search or any search?

Check splunkd.log and /var/log/messages for additional information.  Open a support ticket.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...