Unable to start splunk, failed with crash report

jitsinha — Tue, 29 Sep 2020 07:22:44 GMT

[build aa7d4b1ccb80] 2015-09-26 11:27:52
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 1039871 running under UID 1838232.
Crashing thread: PipelineAllocatorThread
Registers:
RIP: [0x00000033F9432625] gsignal + 53 (/lib64/libc.so.6)
RDI: [0x00000000000FDDFF]
RSI: [0x00000000000FDE71]
RBP: [0x0000000001F2DDA0]
RSP: [0x00007FE0405FDAD8]
RAX: [0x0000000000000000]
RBX: [0x00007FE03FC2C178]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x000000000000000A]
R9: [0x00007FE0405FE700]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x00007FE03FC3E040]
R13: [0x00007FE03FC1E2B8]
R14: [0x00000000018100CC]
R15: [0x00007FE0510A4950]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace:
[0x00000033F9432625] gsignal + 53 (/lib64/libc.so.6)
[0x00000033F9433E05] abort + 373 (/lib64/libc.so.6)
[0x00000000017FD12D] ZN9gnu_cxx27verbose_terminate_handlerEv + 285 (splunkd)
[0x00000000017B76D6] _ZN10cxxabiv111_terminateEPFvvE + 6 (splunkd)
[0x00000000017B7703] ? (splunkd)
[0x00000000017B898E] ? (splunkd)
[0x00000000010A016A] ? (splunkd)
[0x000000000109FCCC] _ZN6ThreadC2EPKcz + 588 (splunkd)
[0x0000000000DB0B17] _ZN8PipelineC2ERK3StrRK15PluginProcessorPK7XmlNodeP13InThreadActorP11PipelineSet + 87 (splunkd)
[0x0000000000DB5775] _ZN23PipelineAllocatorThread4mainEv + 85 (splunkd)
[0x000000000109F0EE] _ZN6Thread8callMainEPv + 62 (splunkd)
[0x00000033F98079D1] ? (/lib64/libpthread.so.0)
[0x00000033F94E88DD] clone + 109 (/lib64/libc.so.6)
Linux / xxx.prod.sin2.secureserver.net / 2.6.32-531.29.2.lve1.3.11.10.el6.x86_64 / #1 SMP Fri Jun 12 15:09:02 EDT 2015 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2015-09-26 11:26:07.692 -0700 splunkd started (build aa7d4b1ccb80)
terminate called after throwing an instance of 'ThreadException'
what(): Main Thread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 45 threads active
2015-09-26 11:27:51.646 -0700 splunkd started (build aa7d4b1ccb80)
terminate called after throwing an instance of 'ThreadException'
what(): PipelineAllocatorThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 42 threads active

glibc version: 2.12
glibc release: stable
Last errno: 12
Threads running: 42
Runtime: 0.864909s
argv: [splunkd -p 8089 start]
Thread: "PipelineAllocatorThread", did_join=1, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7fff28fe9080:
00000000 00 e7 5f 40 e0 7f 00 00 |.._@....|
00000008

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 000306E4 00200800 7FBEE3FF BFEBFBFF
2: 76036301 00F0B2FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000040 00000040 00000003 00001120
6: 00000077 00000002 00000009 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000001 00000000 00000000 00000000
A: 07300403 00000000 00000000 00000603
B: 00000000 00000000 000000FD 00000000
C: 00000000 00000000 00000000 00000000
😧 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 2C100800
80000002: 20202020 746E4920 52286C65 65582029
80000003: 52286E6F 50432029 35452055 3336322D
80000004: 76204C30 20402032 30342E32 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 0000302E 00000000 00000000 00000000
terminating...

Any help!!

Re: Unable to start splunk, failed with crash report

scruse — Sat, 26 Sep 2015 20:47:59 GMT

Seeing anything in splunkd.log?

This appears to be an issue with threads being unable to be opened from your crash report.

42/45 doesn't seem inordinately high, but i would recommend opening a support ticket to see if there isn't something bigger wrong.

one other thing you may want to look at is your ulimits for processes and files for the user splunk is running as.

Re: Unable to start splunk, failed with crash report

jitsinha — Tue, 29 Sep 2020 07:29:33 GMT

index= earliest=-2d@d | stats count by host, _time | join host [search index= earliest=-2d@d | stats latest(_time) as latesttime by host | table host, latesttime] | eval lastTwoHrCount = if(_time<=now() AND _time>=now()-7200,count,0) | eval lastHrCountFrmLatestEvent = if(_time<=now() AND _time>=now()-7200,count,0) | stats sum(lastTwoHrCount) as lastTwoHrCount, max(lastHrCountFrmLatestEvent) as lastHrCountFrmLatestEvent by host | eval status = if(lastTwoHrCount==0 AND lastHrCountFrmLatestEvent<5000 ,"DOWN","UP")
| table host,lastTwoHrCount,lastHrCountFrmLatestEvent,status
| rename host as Host, lastTwoHrCount as "Last 2hrs Count", lastHrCountFrmLatestEvent as "Last Hour Count from LatestEvent", status as Status | fillnull

topic Re: Unable to start splunk, failed with crash report in Splunk Search

Unable to start splunk, failed with crash report

Re: Unable to start splunk, failed with crash report

Re: Unable to start splunk, failed with crash report