Splunk thread segmentation Fault

[mdegann]

Whenever i try to do a spark line with a certain amount of data the thread crashes and the search doesn't finish. I'm using Splunk 6.0.

[build 182037] 2014-04-08 17:40:35
Received fatal signal 11 (Segmentation fault).
 Cause:
   No memory mapped at address [0x00007F05D54F2F40].
 Crashing thread: dispatch
 Registers:
    RIP:  [0x00007F051C0E6BC6] free + 598 (/opt/splunk/lib/libjemalloc.so.1)
    RDI:  [0x00007F0517806000]
    RSI:  [0x0000000000000000]
    RBP:  [0x00007F0517808008]
    RSP:  [0x00007F05183FC6F0]
    RAX:  [0x00007F05183FF6E0]
    RBX:  [0x00007F05175B7100]
    RCX:  [0x00007F051C129FB0]
    RDX:  [0x0000000017B37701]
    R8:  [0x0000000000000000]
    R9:  [0x00007F051C129F60]
    R10:  [0x00007F051A800000]
    R11:  [0x00007F0517B37740]
    R12:  [0x0000000000002000]
    R13:  [0x00007F0517806000]
    R14:  [0x00000000000000DE]
    R15:  [0x0000000017B37700]
    EFL:  [0x0000000000010206]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000006]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]

 OS: Linux
 Arch: x86-64

 Backtrace:
  [0x00007F051C0E6BC6] free + 598 (/opt/splunk/lib/libjemalloc.so.1)
  [0x000000000105888D] _ZNSt8_Rb_treeI3StrSt4pairIKS0_16StatsInclOptionsESt10_Select1stIS4_ESt4lessIS0_ESaIS4_EE8_M_eraseEPSt13_Rb_tree_nodeIS4_E + 77 ([splunkd)
  [0x00000000010588F5] _ZN13ScopedPointerISt3mapI3Str16StatsInclOptionsSt4lessIS1_ESaISt4pairIKS1_S2_EEEED2Ev + 21 ([splunkd)
  [0x000000000104753A] _ZNK14StatsProcessor15output_internalEmR13SearchResultsR17SearchResultsInfobbb + 330 ([splunkd)
  [0x0000000001056E14] _ZN14StatsProcessor15reduce_and_emitER13SearchResultsR17SearchResultsInfob + 1956 ([splunkd)
  [0x0000000000FF2A72] _ZN17NewChartProcessor14execute_outputER13SearchResultsR17SearchResultsInfo + 2530 ([splunkd)
  [0x0000000000FFBE55] _ZN17NewChartProcessor31execute_output_and_set_metadataER13SearchResultsR17SearchResultsInfo + 69 ([splunkd)
  [0x0000000000F00DE3] _ZN14DispatchThread15generatePreviewERK8Pathname + 3347 ([splunkd)
  [0x0000000000F013EF] _ZN14DispatchThread15generatePreviewERK8PathnameRK3Str + 47 ([splunkd)
  [0x0000000000F04218] _ZN14DispatchThread8mainImplEv + 11656 ([splunkd)
  [0x0000000000F0F0EF] _ZN14DispatchThread4mainEv + 223 ([splunkd)
  [0x0000000000D81A2D] _ZN6Thread8callMainEPv + 61 ([splunkd)
  [0x00007F051B132E9A] ? (/lib/x86_64-linux-gnu/libpthread.so.0)
  [0x00007F051AE5ECCD] clone + 109 (/lib/x86_64-linux-gnu/libc.so.6)
 Linux / splunk01 / 3.5.0-36-generic / #57-Ubuntu SMP Wed Jun 19 15:10:49 UTC 2013 / x86_64
 /etc/debian_version: wheezy/sid
 MAP: 00400000-017a5000 r-xp 00000000 fc:00 25431840                           /opt/splunk/bin/splunkd
 MAP: 017a5000-017b9000 rw-p 013a5000 fc:00 25431840                           /opt/splunk/bin/splunkd
 MAP: 017b9000-017f4000 rw-p 00000000 00:00 0                                  [heap]
 MAP: 7f05155ea000-7f05155ff000 r-xp 00000000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f05155ff000-7f05157fe000 ---p 00015000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f05157fe000-7f05157ff000 r--p 00014000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f05157ff000-7f0515800000 rw-p 00015000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f0515800000-7f0516c00000 rw-p 00000000 00:00 0 
 MAP: 7f0516dfd000-7f0516dfe000 ---p 00000000 00:00 0 
 MAP: 7f0516dfe000-7f0516ffe000 rw-p 00000000 00:00 0                          [stack:22393]
 MAP: 7f0516ffe000-7f0516fff000 ---p 00000000 00:00 0 
 MAP: 7f0516fff000-7f05171ff000 rw-p 00000000 00:00 0 
 MAP: 7f05171ff000-7f0517200000 ---p 00000000 00:00 0 
 MAP: 7f0517200000-7f0517c00000 rw-p 00000000 00:00 0 
 MAP: 7f0517dfe000-7f0517dff000 ---p 00000000 00:00 0 
 MAP: 7f0517dff000-7f0517fff000 rw-p 00000000 00:00 0 
 MAP: 7f0517fff000-7f0518000000 ---p 00000000 00:00 0 
 MAP: 7f0518000000-7f0519000000 rw-p 00000000 00:00 0                          [stack:20998]
 MAP: 7f05191ff000-7f0519200000 ---p 00000000 00:00 0 
 MAP: 7f0519200000-7f0519800000 rw-p 00000000 00:00 0                          [stack:20972]
 MAP: 7f05199fd000-7f05199fe000 ---p 00000000 00:00 0 
 MAP: 7f05199fe000-7f0519bfe000 rw-p 00000000 00:00 0                          [stack:20971]
 MAP: 7f0519bfe000-7f0519bff000 ---p 00000000 00:00 0 
 MAP: 7f0519bff000-7f0519dff000 rw-p 00000000 00:00 0                          [stack:20970]
 MAP: 7f0519dff000-7f0519e00000 ---p 00000000 00:00 0 
 MAP: 7f0519e00000-7f051a000000 rw-p 00000000 00:00 0                          [stack:20968]
 MAP: 7f051a000000-7f051a400000 rw-p 00000000 00:00 0 
 MAP: 7f051a400000-7f051ac00000 rw-p 00000000 00:00 0 
 MAP: 7f051ad5c000-7f051ad5d000 ---p 00000000 00:00 0 
 MAP: 7f051ad5d000-7f051ad66000 rw-p 00000000 00:00 0 
 MAP: 7f051ad66000-7f051ad67000 ---p 00000000 00:00 0 
 MAP: 7f051ad67000-7f051ad6b000 rw-p 00000000 00:00 0 
 MAP: 7f051ad6b000-7f051af20000 r-xp 00000000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051af20000-7f051b11f000 ---p 001b5000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051b11f000-7f051b123000 r--p 001b4000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051b123000-7f051b125000 rw-p 001b8000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051b125000-7f051b12b000 rw-p 00000000 00:00 0 
 MAP: 7f051b12b000-7f051b143000 r-xp 00000000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b143000-7f051b342000 ---p 00018000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b342000-7f051b343000 r--p 00017000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b343000-7f051b344000 rw-p 00018000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b344000-7f051b348000 rw-p 00000000 00:00 0 
 MAP: 7f051b348000-7f051b443000 r-xp 00000000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b443000-7f051b642000 ---p 000fb000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b642000-7f051b643000 r--p 000fa000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b643000-7f051b644000 rw-p 000fb000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b644000-7f051b65b000 r-xp 00000000 fc:00 25429406                   /opt/splunk/lib/libz.so.1.2.8
 MAP: 7f051b65b000-7f051b65c000 rw-p 00016000 fc:00 25429406                   /opt/splunk/lib/libz.so.1.2.8
 MAP: 7f051b65c000-7f051b6e8000 r-xp 00000000 fc:00 25431661                   /opt/splunk/lib/libsqlite3.so.0.8.6
 MAP: 7f051b6e8000-7f051b6eb000 rw-p 0008b000 fc:00 25431661                   /opt/splunk/lib/libsqlite3.so.0.8.6
 MAP: 7f051b6eb000-7f051b6ed000 r-xp 00000000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b6ed000-7f051b8ed000 ---p 00002000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b8ed000-7f051b8ee000 r--p 00002000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b8ee000-7f051b8ef000 rw-p 00003000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b8ef000-7f051bb18000 r-xp 00000000 fc:00 25429404                   /opt/splunk/lib/libcrypto.so.1.0.0
 MAP: 7f051bb18000-7f051bb42000 rw-p 00228000 fc:00 25429404                   /opt/splunk/lib/libcrypto.so.1.0.0
 MAP: 7f051bb42000-7f051bb46000 rw-p 00000000 00:00 0 
 MAP: 7f051bb46000-7f051bcf7000 r-xp 00000000 fc:00 25431658                   /opt/splunk/lib/libxml2.so.2.9.1
 MAP: 7f051bcf7000-7f051bd01000 rw-p 001b1000 fc:00 25431658                   /opt/splunk/lib/libxml2.so.2.9.1
 MAP: 7f051bd01000-7f051bd02000 rw-p 00000000 00:00 0 
 MAP: 7f051bd02000-7f051bd09000 r-xp 00000000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bd09000-7f051bf08000 ---p 00007000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bf08000-7f051bf09000 r--p 00006000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bf09000-7f051bf0a000 rw-p 00007000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bf0a000-7f051bf2c000 r-xp 00000000 fc:00 9440961                    /lib/x86_64-linux-gnu/ld-2.15.so
 MAP: 7f051bf36000-7f051bf37000 rw-p 00000000 00:00 0 
 MAP: 7f051bf37000-7f051bf46000 r-xp 00000000 fc:00 25429410                   /opt/splunk/lib/libbz2.so.1.0.3
 MAP: 7f051bf46000-7f051bf48000 rw-p 0000e000 fc:00 25429410                   /opt/splunk/lib/libbz2.so.1.0.3
 MAP: 7f051bf48000-7f051bfd7000 r-xp 00000000 fc:00 25429401                   /opt/splunk/lib/libarchive.so.13.1.2
 MAP: 7f051bfd7000-7f051bfdb000 rw-p 0008e000 fc:00 25429401                   /opt/splunk/lib/libarchive.so.13.1.2
 MAP: 7f051bfdb000-7f051bfdd000 rw-p 00000000 00:00 0 
 MAP: 7f051bfdd000-7f051c03e000 r-xp 00000000 fc:00 25431660                   /opt/splunk/lib/libssl.so.1.0.0
 MAP: 7f051c03e000-7f051c048000 rw-p 00061000 fc:00 25431660                   /opt/splunk/lib/libssl.so.1.0.0
 MAP: 7f051c048000-7f051c049000 rw-p 00000000 00:00 0 
 MAP: 7f051c049000-7f051c097000 r-xp 00000000 fc:00 25431659                   /opt/splunk/lib/libxslt.so.1.1.28
 MAP: 7f051c097000-7f051c099000 rw-p 0004d000 fc:00 25431659                   /opt/splunk/lib/libxslt.so.1.1.28
 MAP: 7f051c099000-7f051c0d9000 r-xp 00000000 fc:00 25431644                   /opt/splunk/lib/libpcre.so.1.2.1
 MAP: 7f051c0d9000-7f051c0da000 rw-p 0003f000 fc:00 25431644                   /opt/splunk/lib/libpcre.so.1.2.1
 MAP: 7f051c0da000-7f051c0db000 rw-p 00000000 00:00 0 
 MAP: 7f051c0e2000-7f051c0e3000 rw-p 00000000 00:00 0 
 MAP: 7f051c0e3000-7f051c116000 r-xp 00000000 fc:00 25431801                   /opt/splunk/lib/libjemalloc.so.1
 MAP: 7f051c116000-7f051c118000 rw-p 00033000 fc:00 25431801                   /opt/splunk/lib/libjemalloc.so.1
 MAP: 7f051c118000-7f051c12c000 rw-p 00000000 00:00 0 
 MAP: 7f051c12c000-7f051c12d000 r--p 00022000 fc:00 9440961                    /lib/x86_64-linux-gnu/ld-2.15.so
 MAP: 7f051c12d000-7f051c12f000 rw-p 00023000 fc:00 9440961                    /lib/x86_64-linux-gnu/ld-2.15.so
 MAP: 7fff53226000-7fff53247000 rw-p 00000000 00:00 0                          [stack]
 MAP: 7fff5325f000-7fff53260000 r-xp 00000000 00:00 0                          [vdso]
 MAP: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Last errno: 0
Threads running: 7
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=29117] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=29117] search --id=1396993215.369 --maxbuckets=0 --ttl=600 --maxout=50000000 --maxtime=8640000 --lookups=1 --reduce_freq=10 --user=mdegan --pro --roles=admin:can_delete:power:user
Thread: "dispatch", did_join=1, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f05185a0010:
00000000  00 f7 3f 18 05 7f 00 00                           |..?.....|
00000008

x86 CPUID registers:
         0: 0000000B 756E6547 6C65746E 49656E69
         1: 00020651 03080800 82982203 1FEBFBFF
         2: 55035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00001000 00001000 00000003 00001120
         6: 00000007 00000001 00000009 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000001 00000000 00000000 00000000
         A: 07300403 0000007F 00000000 00000000
         B: 00000000 00000000 1B139CCD 00000003
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 6E6F6558 20295228 20555043 202D3745
  80000004: 30373832 20402020 30342E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003028 00000000 00000000 00000000
terminating...

[mrgibbon]

I managed to work around this by un-taring the current version of Splunk over the top of the installation.
Running a chown command to make sure the files were all owned by the right user, then starting up again.
Worked for me, hope this can help someone else.

[kamal_jagga]

Would you be able to advise the directories/files whose permissions you changed. This happened on 3 out of 8 of my indexers with a fraction of minute.