- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk thread segmentation Fault
mdegann
Explorer
04-08-2014
02:55 PM
Whenever i try to do a spark line with a certain amount of data the thread crashes and the search doesn't finish. I'm using Splunk 6.0.
[build 182037] 2014-04-08 17:40:35
Received fatal signal 11 (Segmentation fault).
Cause:
No memory mapped at address [0x00007F05D54F2F40].
Crashing thread: dispatch
Registers:
RIP: [0x00007F051C0E6BC6] free + 598 (/opt/splunk/lib/libjemalloc.so.1)
RDI: [0x00007F0517806000]
RSI: [0x0000000000000000]
RBP: [0x00007F0517808008]
RSP: [0x00007F05183FC6F0]
RAX: [0x00007F05183FF6E0]
RBX: [0x00007F05175B7100]
RCX: [0x00007F051C129FB0]
RDX: [0x0000000017B37701]
R8: [0x0000000000000000]
R9: [0x00007F051C129F60]
R10: [0x00007F051A800000]
R11: [0x00007F0517B37740]
R12: [0x0000000000002000]
R13: [0x00007F0517806000]
R14: [0x00000000000000DE]
R15: [0x0000000017B37700]
EFL: [0x0000000000010206]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000006]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]
OS: Linux
Arch: x86-64
Backtrace:
[0x00007F051C0E6BC6] free + 598 (/opt/splunk/lib/libjemalloc.so.1)
[0x000000000105888D] _ZNSt8_Rb_treeI3StrSt4pairIKS0_16StatsInclOptionsESt10_Select1stIS4_ESt4lessIS0_ESaIS4_EE8_M_eraseEPSt13_Rb_tree_nodeIS4_E + 77 ([splunkd)
[0x00000000010588F5] _ZN13ScopedPointerISt3mapI3Str16StatsInclOptionsSt4lessIS1_ESaISt4pairIKS1_S2_EEEED2Ev + 21 ([splunkd)
[0x000000000104753A] _ZNK14StatsProcessor15output_internalEmR13SearchResultsR17SearchResultsInfobbb + 330 ([splunkd)
[0x0000000001056E14] _ZN14StatsProcessor15reduce_and_emitER13SearchResultsR17SearchResultsInfob + 1956 ([splunkd)
[0x0000000000FF2A72] _ZN17NewChartProcessor14execute_outputER13SearchResultsR17SearchResultsInfo + 2530 ([splunkd)
[0x0000000000FFBE55] _ZN17NewChartProcessor31execute_output_and_set_metadataER13SearchResultsR17SearchResultsInfo + 69 ([splunkd)
[0x0000000000F00DE3] _ZN14DispatchThread15generatePreviewERK8Pathname + 3347 ([splunkd)
[0x0000000000F013EF] _ZN14DispatchThread15generatePreviewERK8PathnameRK3Str + 47 ([splunkd)
[0x0000000000F04218] _ZN14DispatchThread8mainImplEv + 11656 ([splunkd)
[0x0000000000F0F0EF] _ZN14DispatchThread4mainEv + 223 ([splunkd)
[0x0000000000D81A2D] _ZN6Thread8callMainEPv + 61 ([splunkd)
[0x00007F051B132E9A] ? (/lib/x86_64-linux-gnu/libpthread.so.0)
[0x00007F051AE5ECCD] clone + 109 (/lib/x86_64-linux-gnu/libc.so.6)
Linux / splunk01 / 3.5.0-36-generic / #57-Ubuntu SMP Wed Jun 19 15:10:49 UTC 2013 / x86_64
/etc/debian_version: wheezy/sid
MAP: 00400000-017a5000 r-xp 00000000 fc:00 25431840 /opt/splunk/bin/splunkd
MAP: 017a5000-017b9000 rw-p 013a5000 fc:00 25431840 /opt/splunk/bin/splunkd
MAP: 017b9000-017f4000 rw-p 00000000 00:00 0 [heap]
MAP: 7f05155ea000-7f05155ff000 r-xp 00000000 fc:00 9437447 /lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7f05155ff000-7f05157fe000 ---p 00015000 fc:00 9437447 /lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7f05157fe000-7f05157ff000 r--p 00014000 fc:00 9437447 /lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7f05157ff000-7f0515800000 rw-p 00015000 fc:00 9437447 /lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7f0515800000-7f0516c00000 rw-p 00000000 00:00 0
MAP: 7f0516dfd000-7f0516dfe000 ---p 00000000 00:00 0
MAP: 7f0516dfe000-7f0516ffe000 rw-p 00000000 00:00 0 [stack:22393]
MAP: 7f0516ffe000-7f0516fff000 ---p 00000000 00:00 0
MAP: 7f0516fff000-7f05171ff000 rw-p 00000000 00:00 0
MAP: 7f05171ff000-7f0517200000 ---p 00000000 00:00 0
MAP: 7f0517200000-7f0517c00000 rw-p 00000000 00:00 0
MAP: 7f0517dfe000-7f0517dff000 ---p 00000000 00:00 0
MAP: 7f0517dff000-7f0517fff000 rw-p 00000000 00:00 0
MAP: 7f0517fff000-7f0518000000 ---p 00000000 00:00 0
MAP: 7f0518000000-7f0519000000 rw-p 00000000 00:00 0 [stack:20998]
MAP: 7f05191ff000-7f0519200000 ---p 00000000 00:00 0
MAP: 7f0519200000-7f0519800000 rw-p 00000000 00:00 0 [stack:20972]
MAP: 7f05199fd000-7f05199fe000 ---p 00000000 00:00 0
MAP: 7f05199fe000-7f0519bfe000 rw-p 00000000 00:00 0 [stack:20971]
MAP: 7f0519bfe000-7f0519bff000 ---p 00000000 00:00 0
MAP: 7f0519bff000-7f0519dff000 rw-p 00000000 00:00 0 [stack:20970]
MAP: 7f0519dff000-7f0519e00000 ---p 00000000 00:00 0
MAP: 7f0519e00000-7f051a000000 rw-p 00000000 00:00 0 [stack:20968]
MAP: 7f051a000000-7f051a400000 rw-p 00000000 00:00 0
MAP: 7f051a400000-7f051ac00000 rw-p 00000000 00:00 0
MAP: 7f051ad5c000-7f051ad5d000 ---p 00000000 00:00 0
MAP: 7f051ad5d000-7f051ad66000 rw-p 00000000 00:00 0
MAP: 7f051ad66000-7f051ad67000 ---p 00000000 00:00 0
MAP: 7f051ad67000-7f051ad6b000 rw-p 00000000 00:00 0
MAP: 7f051ad6b000-7f051af20000 r-xp 00000000 fc:00 9440973 /lib/x86_64-linux-gnu/libc-2.15.so
MAP: 7f051af20000-7f051b11f000 ---p 001b5000 fc:00 9440973 /lib/x86_64-linux-gnu/libc-2.15.so
MAP: 7f051b11f000-7f051b123000 r--p 001b4000 fc:00 9440973 /lib/x86_64-linux-gnu/libc-2.15.so
MAP: 7f051b123000-7f051b125000 rw-p 001b8000 fc:00 9440973 /lib/x86_64-linux-gnu/libc-2.15.so
MAP: 7f051b125000-7f051b12b000 rw-p 00000000 00:00 0
MAP: 7f051b12b000-7f051b143000 r-xp 00000000 fc:00 9440956 /lib/x86_64-linux-gnu/libpthread-2.15.so
MAP: 7f051b143000-7f051b342000 ---p 00018000 fc:00 9440956 /lib/x86_64-linux-gnu/libpthread-2.15.so
MAP: 7f051b342000-7f051b343000 r--p 00017000 fc:00 9440956 /lib/x86_64-linux-gnu/libpthread-2.15.so
MAP: 7f051b343000-7f051b344000 rw-p 00018000 fc:00 9440956 /lib/x86_64-linux-gnu/libpthread-2.15.so
MAP: 7f051b344000-7f051b348000 rw-p 00000000 00:00 0
MAP: 7f051b348000-7f051b443000 r-xp 00000000 fc:00 9440969 /lib/x86_64-linux-gnu/libm-2.15.so
MAP: 7f051b443000-7f051b642000 ---p 000fb000 fc:00 9440969 /lib/x86_64-linux-gnu/libm-2.15.so
MAP: 7f051b642000-7f051b643000 r--p 000fa000 fc:00 9440969 /lib/x86_64-linux-gnu/libm-2.15.so
MAP: 7f051b643000-7f051b644000 rw-p 000fb000 fc:00 9440969 /lib/x86_64-linux-gnu/libm-2.15.so
MAP: 7f051b644000-7f051b65b000 r-xp 00000000 fc:00 25429406 /opt/splunk/lib/libz.so.1.2.8
MAP: 7f051b65b000-7f051b65c000 rw-p 00016000 fc:00 25429406 /opt/splunk/lib/libz.so.1.2.8
MAP: 7f051b65c000-7f051b6e8000 r-xp 00000000 fc:00 25431661 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f051b6e8000-7f051b6eb000 rw-p 0008b000 fc:00 25431661 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f051b6eb000-7f051b6ed000 r-xp 00000000 fc:00 9440971 /lib/x86_64-linux-gnu/libdl-2.15.so
MAP: 7f051b6ed000-7f051b8ed000 ---p 00002000 fc:00 9440971 /lib/x86_64-linux-gnu/libdl-2.15.so
MAP: 7f051b8ed000-7f051b8ee000 r--p 00002000 fc:00 9440971 /lib/x86_64-linux-gnu/libdl-2.15.so
MAP: 7f051b8ee000-7f051b8ef000 rw-p 00003000 fc:00 9440971 /lib/x86_64-linux-gnu/libdl-2.15.so
MAP: 7f051b8ef000-7f051bb18000 r-xp 00000000 fc:00 25429404 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f051bb18000-7f051bb42000 rw-p 00228000 fc:00 25429404 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f051bb42000-7f051bb46000 rw-p 00000000 00:00 0
MAP: 7f051bb46000-7f051bcf7000 r-xp 00000000 fc:00 25431658 /opt/splunk/lib/libxml2.so.2.9.1
MAP: 7f051bcf7000-7f051bd01000 rw-p 001b1000 fc:00 25431658 /opt/splunk/lib/libxml2.so.2.9.1
MAP: 7f051bd01000-7f051bd02000 rw-p 00000000 00:00 0
MAP: 7f051bd02000-7f051bd09000 r-xp 00000000 fc:00 9440960 /lib/x86_64-linux-gnu/librt-2.15.so
MAP: 7f051bd09000-7f051bf08000 ---p 00007000 fc:00 9440960 /lib/x86_64-linux-gnu/librt-2.15.so
MAP: 7f051bf08000-7f051bf09000 r--p 00006000 fc:00 9440960 /lib/x86_64-linux-gnu/librt-2.15.so
MAP: 7f051bf09000-7f051bf0a000 rw-p 00007000 fc:00 9440960 /lib/x86_64-linux-gnu/librt-2.15.so
MAP: 7f051bf0a000-7f051bf2c000 r-xp 00000000 fc:00 9440961 /lib/x86_64-linux-gnu/ld-2.15.so
MAP: 7f051bf36000-7f051bf37000 rw-p 00000000 00:00 0
MAP: 7f051bf37000-7f051bf46000 r-xp 00000000 fc:00 25429410 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f051bf46000-7f051bf48000 rw-p 0000e000 fc:00 25429410 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f051bf48000-7f051bfd7000 r-xp 00000000 fc:00 25429401 /opt/splunk/lib/libarchive.so.13.1.2
MAP: 7f051bfd7000-7f051bfdb000 rw-p 0008e000 fc:00 25429401 /opt/splunk/lib/libarchive.so.13.1.2
MAP: 7f051bfdb000-7f051bfdd000 rw-p 00000000 00:00 0
MAP: 7f051bfdd000-7f051c03e000 r-xp 00000000 fc:00 25431660 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f051c03e000-7f051c048000 rw-p 00061000 fc:00 25431660 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f051c048000-7f051c049000 rw-p 00000000 00:00 0
MAP: 7f051c049000-7f051c097000 r-xp 00000000 fc:00 25431659 /opt/splunk/lib/libxslt.so.1.1.28
MAP: 7f051c097000-7f051c099000 rw-p 0004d000 fc:00 25431659 /opt/splunk/lib/libxslt.so.1.1.28
MAP: 7f051c099000-7f051c0d9000 r-xp 00000000 fc:00 25431644 /opt/splunk/lib/libpcre.so.1.2.1
MAP: 7f051c0d9000-7f051c0da000 rw-p 0003f000 fc:00 25431644 /opt/splunk/lib/libpcre.so.1.2.1
MAP: 7f051c0da000-7f051c0db000 rw-p 00000000 00:00 0
MAP: 7f051c0e2000-7f051c0e3000 rw-p 00000000 00:00 0
MAP: 7f051c0e3000-7f051c116000 r-xp 00000000 fc:00 25431801 /opt/splunk/lib/libjemalloc.so.1
MAP: 7f051c116000-7f051c118000 rw-p 00033000 fc:00 25431801 /opt/splunk/lib/libjemalloc.so.1
MAP: 7f051c118000-7f051c12c000 rw-p 00000000 00:00 0
MAP: 7f051c12c000-7f051c12d000 r--p 00022000 fc:00 9440961 /lib/x86_64-linux-gnu/ld-2.15.so
MAP: 7f051c12d000-7f051c12f000 rw-p 00023000 fc:00 9440961 /lib/x86_64-linux-gnu/ld-2.15.so
MAP: 7fff53226000-7fff53247000 rw-p 00000000 00:00 0 [stack]
MAP: 7fff5325f000-7fff53260000 r-xp 00000000 00:00 0 [vdso]
MAP: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Last errno: 0
Threads running: 7
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=29117] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=29117] search --id=1396993215.369 --maxbuckets=0 --ttl=600 --maxout=50000000 --maxtime=8640000 --lookups=1 --reduce_freq=10 --user=mdegan --pro --roles=admin:can_delete:power:user
Thread: "dispatch", did_join=1, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f05185a0010:
00000000 00 f7 3f 18 05 7f 00 00 |..?.....|
00000008
x86 CPUID registers:
0: 0000000B 756E6547 6C65746E 49656E69
1: 00020651 03080800 82982203 1FEBFBFF
2: 55035A01 00F0B2FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00001000 00001000 00000003 00001120
6: 00000007 00000001 00000009 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000001 00000000 00000000 00000000
A: 07300403 0000007F 00000000 00000000
B: 00000000 00000000 1B139CCD 00000003
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 28100800
80000002: 20202020 49202020 6C65746E 20295228
80000003: 6E6F6558 20295228 20555043 202D3745
80000004: 30373832 20402020 30342E32 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
mrgibbon
Contributor
10-05-2016
05:04 PM
I managed to work around this by un-taring the current version of Splunk over the top of the installation.
Running a chown command to make sure the files were all owned by the right user, then starting up again.
Worked for me, hope this can help someone else.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
kamal_jagga
Contributor
06-07-2018
10:05 PM
Would you be able to advise the directories/files whose permissions you changed. This happened on 3 out of 8 of my indexers with a fraction of minute.
