Why did Crash log occur?

[khyoung7410]

[build 249101] 2017-08-12 20:00:03
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: BucketSummaryActorThread
Registers:
RIP: [0x00007F906C264421] ? (/opt/splunk/lib/libjemalloc.so.1)
RDI: [0xBCA1AF286BCA1AF4]
RSI: [0x0000000000000400]
RBP: [0x00007F906C29BF80]
RSP: [0x00007F9069FF8DC0]
RAX: [0x0000000000000079]
RBX: [0x00007F906C29BFB0]
RCX: [0x0000000000079000]
RDX: [0x35E50D79435E50DC]
R8: [0x86BCA1AF286BCA1B]
R9: [0x9435E50D79435E64]
R10: [0x00007F906C29B2E0]
R11: [0x0000000000000021]
R12: [0x00007F9068400AF8]
R13: [0x0000000000000001]
R14: [0x00007F90684002A0]
R15: [0x00007F9068400000]
EFL: [0x0000000000010A03]
TRAPNO: [0x000000000000000D]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace:
[0x00007F906C264421] ? (/opt/splunk/lib/libjemalloc.so.1)
[0x00007F906C267547] ? (/opt/splunk/lib/libjemalloc.so.1)
[0x00000000012D2DC5] ? ([splunkd)
[0x00000000012D3B60] ? ([splunkd)
[0x00000000012D54C7] st_tsidx_apply + 23 ([splunkd)
[0x00000000012D5A49] st_tsidx_query + 153 ([splunkd)
[0x00000000012D19E6] st_query + 166 ([splunkd)
[0x0000000000F6C068] ZN5STMgr8STHandle5queryERK13st_query_specP8st_tsvaljjb + 56 ([splunkd)
[0x0000000000971403] _ZN24DatabaseDirectoryManager6Bucket5queryEP13st_query_specRSt6vectorI12ValueWrapperSaIS4_EEjbRK9StrVector + 899 ([splunkd)
[0x0000000000A02E62] _ZN17IndexScopedSearch19fetchValuesFromDiskERSt6vectorI12ValueWrapperSaIS1_EEmmRb + 850 ([splunkd)
[0x0000000000A0350B] _ZN17IndexScopedSearch14prefetchValuesEm + 251 ([splunkd)
[0x00000000009FD6EE] _ZN14CursoredSearch23fetchResultsFromIndexesER13SearchResultsR17SearchResultsInfomb + 622 ([splunkd)
[0x00000000009FE027] _ZN14CursoredSearch12fetchResultsER13SearchResultsR17SearchResultsInfommb + 1399 ([splunkd)
[0x00000000009F3ADB] _ZN14SearchOperator16historicalSearchER13SearchResultsR17SearchResultsInfo + 2427 ([splunkd)
[0x00000000009F4463] _ZN14SearchOperator7executeER13SearchResultsR17SearchResultsInfo + 1139 ([splunkd)
[0x0000000000F6573A] _ZN15SearchProcessor16execute_dispatchER13SearchResultsR17SearchResultsInfoRK3Str + 778 ([splunkd)
[0x0000000000F60AF4] _ZN14SearchPipeline7executeER13SearchResultsR17SearchResultsInfo + 180 ([splunkd)
[0x0000000000A1ACF7] _ZN13BucketSummary6createEP14SearchPipelineRK17SearchResultsInfoRKNS_13CreateOptionsE14TimeBucketSpanS1_m + 1895 ([splunkd)
[0x0000000000A1BF04] _ZN13BucketSummary6createEP14SearchPipelineRK17SearchResultsInfoRKNS_13CreateOptionsERK9StrVectorS1 + 1188 ([splunkd)
[0x0000000000A21769] ZN24BucketSummaryActorThread4mainEv + 873 ([splunkd)
[0x0000000000E20E7E] _ZN6Thread8callMainEPv + 62 ([splunkd)
[0x00000038F3A07AA1] ? (/lib64/libpthread.so.0)
[0x00000038F36E8AAD] clone + 109 (/lib64/libc.so.6)
Linux / splunkindex1 / 2.6.32-642.el6.x86_64 / #1 SMP Wed Apr 13 00:51:26 EDT 2016 / x86_64
/etc/redhat-release: Red Hat Enterprise Linux Server release 6.8 (Santiago)
glibc version: 2.12
glibc release: stable
MAP: 00400000-018cd000 r-xp 00000000 08:03 1838359 /opt/splunk/bin/splunkd
MAP: 018cd000-018d8000 rw-p 014cd000 08:03 1838359 /opt/splunk/bin/splunkd
MAP: 018d8000-01912000 rw-p 00000000 00:00 0
MAP: 32e5200000-32e5216000 r-xp 00000000 08:05 262238 /lib64/libgcc_s-4.4.7-20120601.so.1
MAP: 32e5216000-32e5415000 ---p 00016000 08:05 262238 /lib64/libgcc_s-4.4.7-20120601.so.1
MAP: 32e5415000-32e5416000 rw-p 00015000 08:05 262238 /lib64/libgcc_s-4.4.7-20120601.so.1
MAP: 38f2e00000-38f2e20000 r-xp 00000000 08:05 266610 /lib64/ld-2.12.so
MAP: 38f301f000-38f3021000 r--p 0001f000 08:05 266610 /lib64/ld-2.12.so
MAP: 38f3021000-38f3022000 rw-p 00021000 08:05 266610 /lib64/ld-2.12.so
MAP: 38f3022000-38f3023000 rw-p 00000000 00:00 0
MAP: 38f3200000-38f3202000 r-xp 00000000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3202000-38f3402000 ---p 00002000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3402000-38f3403000 r--p 00002000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3403000-38f3404000 rw-p 00003000 08:05 266617 /lib64/libdl-2.12.so
MAP: 38f3600000-38f378a000 r-xp 00000000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f378a000-38f398a000 ---p 0018a000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f398a000-38f398e000 r--p 0018a000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f398e000-38f3990000 rw-p 0018e000 08:05 266611 /lib64/libc-2.12.so
MAP: 38f3990000-38f3994000 rw-p 00000000 00:00 0
MAP: 38f3a00000-38f3a17000 r-xp 00000000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3a17000-38f3c17000 ---p 00017000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3c17000-38f3c18000 r--p 00017000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3c18000-38f3c19000 rw-p 00018000 08:05 266612 /lib64/libpthread-2.12.so
MAP: 38f3c19000-38f3c1d000 rw-p 00000000 00:00 0
MAP: 38f4200000-38f4283000 r-xp 00000000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4283000-38f4482000 ---p 00083000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4482000-38f4483000 r--p 00082000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4483000-38f4484000 rw-p 00083000 08:05 266622 /lib64/libm-2.12.so
MAP: 38f4600000-38f4607000 r-xp 00000000 08:05 266613 /lib64/librt-2.12.so
MAP: 38f4607000-38f4806000 ---p 00007000 08:05 266613 /lib64/librt-2.12.so
MAP: 38f4806000-38f4807000 r--p 00006000 08:05 266613 /lib64/librt-2.12.so
MAP: 38f4807000-38f4808000 rw-p 00007000 08:05 266613 /lib64/librt-2.12.so
MAP: 7f906777a000-7f9067800000 r--s 00000000 fd:08 1149280 /data/splunk/var/lib/splunk/_internaldb/db/hot_v1_1255/1502535344-1502534752-6236436154434463621.tsidx
MAP: 7f9067800000-7f9069c00000 rw-p 00000000 00:00 0
MAP: 7f9069dfd000-7f9069dfe000 ---p 00000000 00:00 0
MAP: 7f9069dfe000-7f9069ffe000 rw-p 00000000 00:00 0
MAP: 7f9069ffe000-7f9069fff000 ---p 00000000 00:00 0
MAP: 7f9069fff000-7f906a1ff000 rw-p 00000000 00:00 0
MAP: 7f906a1ff000-7f906a200000 ---p 00000000 00:00 0
MAP: 7f906a200000-7f906a800000 rw-p 00000000 00:00 0
MAP: 7f906a9ff000-7f906aa00000 ---p 00000000 00:00 0
MAP: 7f906aa00000-7f906b000000 rw-p 00000000 00:00 0
MAP: 7f906b000000-7f906b800000 rw-p 00000000 00:00 0
MAP: 7f906b9ce000-7f906b9cf000 ---p 00000000 00:00 0
MAP: 7f906b9cf000-7f906bbcf000 rw-p 00000000 00:00 0
MAP: 7f906bbcf000-7f906bbd4000 rw-p 00000000 00:00 0
MAP: 7f906bbd4000-7f906bbeb000 r-xp 00000000 08:03 1835039 /opt/splunk/lib/libz.so.1.2.8
MAP: 7f906bbeb000-7f906bbec000 rw-p 00016000 08:03 1835039 /opt/splunk/lib/libz.so.1.2.8
MAP: 7f906bbec000-7f906bc78000 r-xp 00000000 08:03 1838139 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f906bc78000-7f906bc7b000 rw-p 0008b000 08:03 1838139 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f906bc7b000-7f906bc7c000 rw-p 00000000 00:00 0
MAP: 7f906bc7c000-7f906bc8b000 r-xp 00000000 08:03 1835043 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f906bc8b000-7f906bc8d000 rw-p 0000e000 08:03 1835043 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f906bc8d000-7f906bd1c000 r-xp 00000000 08:03 1835034 /opt/splunk/lib/libarchive.so.13.1.2
MAP: 7f906bd1c000-7f906bd20000 rw-p 0008e000 08:03 1835034 /opt/splunk/lib/libarchive.so.13.1.2
MAP: 7f906bd20000-7f906bd22000 rw-p 00000000 00:00 0
MAP: 7f906bd22000-7f906bf4c000 r-xp 00000000 08:03 1835037 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f906bf4c000-7f906bf76000 rw-p 0022a000 08:03 1835037 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f906bf76000-7f906bf7a000 rw-p 00000000 00:00 0
MAP: 7f906bf7a000-7f906bfdd000 r-xp 00000000 08:03 1838138 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f906bfdd000-7f906bfe7000 rw-p 00062000 08:03 1838138 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f906bfe7000-7f906c035000 r-xp 00000000 08:03 1838137 /opt/splunk/lib/libxslt.so.1.1.28
MAP: 7f906c035000-7f906c037000 rw-p 0004d000 08:03 1838137 /opt/splunk/lib/libxslt.so.1.1.28
MAP: 7f906c037000-7f906c038000 rw-p 00000000 00:00 0
MAP: 7f906c038000-7f906c1e9000 r-xp 00000000 08:03 1838136 /opt/splunk/lib/libxml2.so.2.9.1
MAP: 7f906c1e9000-7f906c1f3000 rw-p 001b1000 08:03 1838136 /opt/splunk/lib/libxml2.so.2.9.1
MAP: 7f906c1f3000-7f906c1f4000 rw-p 00000000 00:00 0
MAP: 7f906c1f4000-7f906c234000 r-xp 00000000 08:03 1838121 /opt/splunk/lib/libpcre.so.1.2.1
MAP: 7f906c234000-7f906c235000 rw-p 0003f000 08:03 1838121 /opt/splunk/lib/libpcre.so.1.2.1
MAP: 7f906c235000-7f906c236000 rw-p 00000000 00:00 0
MAP: 7f906c245000-7f906c249000 rw-p 00000000 00:00 0
MAP: 7f906c249000-7f906c24a000 ---p 00000000 00:00 0
MAP: 7f906c24a000-7f906c253000 rw-p 00000000 00:00 0
MAP: 7f906c253000-7f906c254000 ---p 00000000 00:00 0
MAP: 7f906c254000-7f906c255000 rw-p 00000000 00:00 0
MAP: 7f906c255000-7f906c288000 r-xp 00000000 08:03 1838311 /opt/splunk/lib/libjemalloc.so.1
MAP: 7f906c288000-7f906c28a000 rw-p 00033000 08:03 1838311 /opt/splunk/lib/libjemalloc.so.1
MAP: 7f906c28a000-7f906c29d000 rw-p 00000000 00:00 0
MAP: 7ffc595b8000-7ffc595cd000 rw-p 00000000 00:00 0 [stack]
MAP: 7ffc595d0000-7ffc595d1000 r-xp 00000000 00:00 0 [vdso]
MAP: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Last errno: 2
Threads running: 6
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=6211] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=6211] search --id=remote_splunksearch3_schedulernobody_c3BsdW5rX2RlcGxveW1lbnRfbW9uaXRvcg_RMD5ece671823803637a_at_1502535600_84498 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:can_delete:power:splunk-system-role:user
Thread: "BucketSummaryActorThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f9068eb1610:
00000000 00 d7 ff 69 90 7f 00 00 |...i....|
00000008

x86 CPUID registers:
0: 0000000B 756E6547 6C65746E 49656E69
1: 000206C2 01200800 029EE3FF BFEBFBFF
2: 55035A01 00F0B2FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000040 00000040 00000003 00001120
6: 00000005 00000002 00000009 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300403 00000004 00000000 00000603
B: 00000000 00000000 000000CD 00000001
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 2C100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 55504320 20202020 20202020 58202020
80000004: 35373635 20402020 37302E33 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...

[jcrabb_splunk]

I don't see any existing bug for the same crash for Splunk 6.1 or specifically 6.1.6 which is what I believe you have. I'd confirm, based on what I am seeing, that you have THP disabled. It is a memory management scheme that can cause problems when left enabled on instances running Splunk. This can include a variety of memory and stability issues and crashes:

• Transparent huge memory pages (THP) and Splunk performance.
• How do I disable Transparent Huge Pages (THP) and confirm that it is disabled?

If that is already disabled, I would upgrade to the latest version of 6.1 which is 6.1.13, to eliminate any potential bugs you may be encountering.

[masonmorales]

We are on v6.5.2 and are also observing this error. I can confirm that we have THP disabled.