Monitoring Splunk

Why did Crash log occur?

Path Finder

[build 249101] 2017-08-12 20:00:03
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: BucketSummaryActorThread
Registers:
RIP: [0x00007F906C264421] ? (/opt/splunk/lib/libjemalloc.so.1)
RDI: [0xBCA1AF286BCA1AF4]
RSI: [0x0000000000000400]
RBP: [0x00007F906C29BF80]
RSP: [0x00007F9069FF8DC0]
RAX: [0x0000000000000079]
RBX: [0x00007F906C29BFB0]
RCX: [0x0000000000079000]
RDX: [0x35E50D79435E50DC]
R8: [0x86BCA1AF286BCA1B]
R9: [0x9435E50D79435E64]
R10: [0x00007F906C29B2E0]
R11: [0x0000000000000021]
R12: [0x00007F9068400AF8]
R13: [0x0000000000000001]
R14: [0x00007F90684002A0]
R15: [0x00007F9068400000]
EFL: [0x0000000000010A03]
TRAPNO: [0x000000000000000D]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace:
[0x00007F906C264421] ? (/opt/splunk/lib/libjemalloc.so.1)
[0x00007F906C267547] ? (/opt/splunk/lib/libjemalloc.so.1)
[0x00000000012D2DC5] ? ([splunkd)
[0x00000000012D3B60] ? ([splunkd)
[0x00000000012D54C7] st_tsidx_apply + 23 ([splunkd)
[0x00000000012D5A49] st_tsidx_query + 153 ([splunkd)
[0x00000000012D19E6] st_query + 166 ([splunkd)
[0x0000000000F6C068] ZN5STMgr8STHandle5queryERK13st_query_specP8st_tsvaljjb + 56 ([splunkd)
[0x0000000000971403] _ZN24DatabaseDirectoryManager6Bucket5queryEP13st_query_specRSt6vectorI12ValueWrapperSaIS4_EEjbRK9StrVector + 899 ([splunkd)
[0x0000000000A02E62] _ZN17IndexScopedSearch19fetchValuesFromDiskERSt6vectorI12ValueWrapperSaIS1_EEmmRb + 850 ([splunkd)
[0x0000000000A0350B] _ZN17IndexScopedSearch14prefetchValuesEm + 251 ([splunkd)
[0x00000000009FD6EE] _ZN14CursoredSearch23fetchResultsFromIndexesER13SearchResultsR17SearchResultsInfomb + 622 ([splunkd)
[0x00000000009FE027] _ZN14CursoredSearch12fetchResultsER13SearchResultsR17SearchResultsInfommb + 1399 ([splunkd)
[0x00000000009F3ADB] _ZN14SearchOperator16historicalSearchER13SearchResultsR17SearchResultsInfo + 2427 ([splunkd)
[0x00000000009F4463] _ZN14SearchOperator7executeER13SearchResultsR17SearchResultsInfo + 1139 ([splunkd)
[0x0000000000F6573A] _ZN15SearchProcessor16execute_dispatchER13SearchResultsR17SearchResultsInfoRK3Str + 778 ([splunkd)
[0x0000000000F60AF4] _ZN14SearchPipeline7executeER13SearchResultsR17SearchResultsInfo + 180 ([splunkd)
[0x0000000000A1ACF7] _ZN13BucketSummary6createEP14SearchPipelineRK17SearchResultsInfoRKNS_13CreateOptionsE14TimeBucketSpanS1_m + 1895 ([splunkd)
[0x0000000000A1BF04] _ZN13BucketSummary6createEP14SearchPipelineRK17SearchResultsInfoRKNS_13CreateOptionsERK9StrVectorS1
+ 1188 ([splunkd)
[0x0000000000A21769] ZN24BucketSummaryActorThread4mainEv + 873 ([splunkd)
[0x0000000000E20E7E] _ZN6Thread8callMainEPv + 62 ([splunkd)
[0x00000038F3A07AA1] ? (/lib64/libpthread.so.0)
[0x00000038F36E8AAD] clone + 109 (/lib64/libc.so.6)
Linux / splunkindex1 / 2.6.32-642.el6.x86_64 / #1 SMP Wed Apr 13 00:51:26 EDT 2016 / x86_64
/etc/redhat-release: Red Hat Enterprise Linux Server release 6.8 (Santiago)
glibc version: 2.12
glibc release: stable
MAP: 00400000-018cd000 r-xp 00000000 08:03 1838359 /opt/splunk/bin/splunkd
MAP: 018cd000-018d8000 rw-p 014cd000 08:03 1838359 /opt/splunk/bin/splunkd
MAP: 018d8000-01912000 rw-p 00000000 00:00 0
MAP: 32e5200000-32e5216000 r-xp 00000000 08:05 262238 /lib64/libgcc_s-4.4.7-20120601.so.1
MAP: 32e5216000-32e5415000 ---p 00016000 08:05 262238 /lib64/libgcc_s-4.4.7-20120601.so.1
MAP: 32e5415000-32e5416000 rw-p 00015000 08:05 262238 /lib64/libgcc_s-4.4.7-20120601.so.1
MAP: 38f2e00000-38f2e20000 r-xp 00000000 08:05 266610 /lib64/ld-2.12.so
MAP: 38f301f000-38f3021000 r--p 0001f000 08:05 266610 /lib64/ld-2.12.so
MAP: 38f3021000-38f3022000 rw-p 00021000 08:05 266610 /lib64/ld-2.12.so
MAP: 38f3022000-38f3023000 rw-p 00000000 00:00 0
MAP: 38f3200000-38f3202000 r-xp 00000000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3202000-38f3402000 ---p 00002000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3402000-38f3403000 r--p 00002000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3403000-38f3404000 rw-p 00003000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3600000-38f378a000 r-xp 00000000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f378a000-38f398a000 ---p 0018a000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f398a000-38f398e000 r--p 0018a000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f398e000-38f3990000 rw-p 0018e000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f3990000-38f3994000 rw-p 00000000 00:00 0
MAP: 38f3a00000-38f3a17000 r-xp 00000000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3a17000-38f3c17000 ---p 00017000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3c17000-38f3c18000 r--p 00017000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3c18000-38f3c19000 rw-p 00018000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3c19000-38f3c1d000 rw-p 00000000 00:00 0
MAP: 38f4200000-38f4283000 r-xp 00000000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4283000-38f4482000 ---p 00083000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4482000-38f4483000 r--p 00082000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4483000-38f4484000 rw-p 00083000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4600000-38f4607000 r-xp 00000000 08:05 266613 /lib64/librt-2.12.so
MAP: 38f4607000-38f4806000 ---p 00007000 08:05 266613 /lib64/librt-2.12.so
MAP: 38f4806000-38f4807000 r--p 00006000 08:05 266613 /lib64/librt-2.12.so
MAP: 38f4807000-38f4808000 rw-p 00007000 08:05 266613 /lib64/librt-2.12.so
MAP: 7f906777a000-7f9067800000 r--s 00000000 fd:08 1149280 /data/splunk/var/lib/splunk/_internaldb/db/hot_v1_1255/1502535344-1502534752-6236436154434463621.tsidx
MAP: 7f9067800000-7f9069c00000 rw-p 00000000 00:00 0
MAP: 7f9069dfd000-7f9069dfe000 ---p 00000000 00:00 0
MAP: 7f9069dfe000-7f9069ffe000 rw-p 00000000 00:00 0
MAP: 7f9069ffe000-7f9069fff000 ---p 00000000 00:00 0
MAP: 7f9069fff000-7f906a1ff000 rw-p 00000000 00:00 0
MAP: 7f906a1ff000-7f906a200000 ---p 00000000 00:00 0
MAP: 7f906a200000-7f906a800000 rw-p 00000000 00:00 0
MAP: 7f906a9ff000-7f906aa00000 ---p 00000000 00:00 0
MAP: 7f906aa00000-7f906b000000 rw-p 00000000 00:00 0
MAP: 7f906b000000-7f906b800000 rw-p 00000000 00:00 0
MAP: 7f906b9ce000-7f906b9cf000 ---p 00000000 00:00 0
MAP: 7f906b9cf000-7f906bbcf000 rw-p 00000000 00:00 0
MAP: 7f906bbcf000-7f906bbd4000 rw-p 00000000 00:00 0
MAP: 7f906bbd4000-7f906bbeb000 r-xp 00000000 08:03 1835039 /opt/splunk/lib/libz.so.1.2.8
MAP: 7f906bbeb000-7f906bbec000 rw-p 00016000 08:03 1835039 /opt/splunk/lib/libz.so.1.2.8
MAP: 7f906bbec000-7f906bc78000 r-xp 00000000 08:03 1838139 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f906bc78000-7f906bc7b000 rw-p 0008b000 08:03 1838139 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f906bc7b000-7f906bc7c000 rw-p 00000000 00:00 0
MAP: 7f906bc7c000-7f906bc8b000 r-xp 00000000 08:03 1835043 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f906bc8b000-7f906bc8d000 rw-p 0000e000 08:03 1835043 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f906bc8d000-7f906bd1c000 r-xp 00000000 08:03 1835034 /opt/splunk/lib/libarchive.so.13.1.2
MAP: 7f906bd1c000-7f906bd20000 rw-p 0008e000 08:03 1835034 /opt/splunk/lib/libarchive.so.13.1.2
MAP: 7f906bd20000-7f906bd22000 rw-p 00000000 00:00 0
MAP: 7f906bd22000-7f906bf4c000 r-xp 00000000 08:03 1835037 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f906bf4c000-7f906bf76000 rw-p 0022a000 08:03 1835037 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f906bf76000-7f906bf7a000 rw-p 00000000 00:00 0
MAP: 7f906bf7a000-7f906bfdd000 r-xp 00000000 08:03 1838138 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f906bfdd000-7f906bfe7000 rw-p 00062000 08:03 1838138 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f906bfe7000-7f906c035000 r-xp 00000000 08:03 1838137 /opt/splunk/lib/libxslt.so.1.1.28
MAP: 7f906c035000-7f906c037000 rw-p 0004d000 08:03 1838137 /opt/splunk/lib/libxslt.so.1.1.28
MAP: 7f906c037000-7f906c038000 rw-p 00000000 00:00 0
MAP: 7f906c038000-7f906c1e9000 r-xp 00000000 08:03 1838136 /opt/splunk/lib/libxml2.so.2.9.1
MAP: 7f906c1e9000-7f906c1f3000 rw-p 001b1000 08:03 1838136 /opt/splunk/lib/libxml2.so.2.9.1
MAP: 7f906c1f3000-7f906c1f4000 rw-p 00000000 00:00 0
MAP: 7f906c1f4000-7f906c234000 r-xp 00000000 08:03 1838121 /opt/splunk/lib/libpcre.so.1.2.1
MAP: 7f906c234000-7f906c235000 rw-p 0003f000 08:03 1838121 /opt/splunk/lib/libpcre.so.1.2.1
MAP: 7f906c235000-7f906c236000 rw-p 00000000 00:00 0
MAP: 7f906c245000-7f906c249000 rw-p 00000000 00:00 0
MAP: 7f906c249000-7f906c24a000 ---p 00000000 00:00 0
MAP: 7f906c24a000-7f906c253000 rw-p 00000000 00:00 0
MAP: 7f906c253000-7f906c254000 ---p 00000000 00:00 0
MAP: 7f906c254000-7f906c255000 rw-p 00000000 00:00 0
MAP: 7f906c255000-7f906c288000 r-xp 00000000 08:03 1838311 /opt/splunk/lib/libjemalloc.so.1
MAP: 7f906c288000-7f906c28a000 rw-p 00033000 08:03 1838311 /opt/splunk/lib/libjemalloc.so.1
MAP: 7f906c28a000-7f906c29d000 rw-p 00000000 00:00 0
MAP: 7ffc595b8000-7ffc595cd000 rw-p 00000000 00:00 0 [stack]
MAP: 7ffc595d0000-7ffc595d1000 r-xp 00000000 00:00 0 [vdso]
MAP: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Last errno: 2
Threads running: 6
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=6211] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=6211] search --id=remote_splunksearch3_scheduler
nobody_c3BsdW5rX2RlcGxveW1lbnRfbW9uaXRvcg_RMD5ece671823803637a_at_1502535600_84498 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:can_delete:power:splunk-system-role:user
Thread: "BucketSummaryActorThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f9068eb1610:
00000000 00 d7 ff 69 90 7f 00 00 |...i....|
00000008

x86 CPUID registers:
0: 0000000B 756E6547 6C65746E 49656E69
1: 000206C2 01200800 029EE3FF BFEBFBFF
2: 55035A01 00F0B2FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000040 00000040 00000003 00001120
6: 00000005 00000002 00000009 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300403 00000004 00000000 00000603
B: 00000000 00000000 000000CD 00000001
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 2C100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 55504320 20202020 20202020 58202020
80000004: 35373635 20402020 37302E33 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...

Tags (2)
0 Karma

Splunk Employee
Splunk Employee

I don't see any existing bug for the same crash for Splunk 6.1 or specifically 6.1.6 which is what I believe you have. I'd confirm, based on what I am seeing, that you have THP disabled. It is a memory management scheme that can cause problems when left enabled on instances running Splunk. This can include a variety of memory and stability issues and crashes:

If that is already disabled, I would upgrade to the latest version of 6.1 which is 6.1.13, to eliminate any potential bugs you may be encountering.

Jacob
Sr. Technical Support Engineer

Influencer

We are on v6.5.2 and are also observing this error. I can confirm that we have THP disabled.

0 Karma