Installation

Splunk thread segmentation Fault

mdegann
Explorer

Whenever i try to do a spark line with a certain amount of data the thread crashes and the search doesn't finish. I'm using Splunk 6.0.

[build 182037] 2014-04-08 17:40:35
Received fatal signal 11 (Segmentation fault).
 Cause:
   No memory mapped at address [0x00007F05D54F2F40].
 Crashing thread: dispatch
 Registers:
    RIP:  [0x00007F051C0E6BC6] free + 598 (/opt/splunk/lib/libjemalloc.so.1)
    RDI:  [0x00007F0517806000]
    RSI:  [0x0000000000000000]
    RBP:  [0x00007F0517808008]
    RSP:  [0x00007F05183FC6F0]
    RAX:  [0x00007F05183FF6E0]
    RBX:  [0x00007F05175B7100]
    RCX:  [0x00007F051C129FB0]
    RDX:  [0x0000000017B37701]
    R8:  [0x0000000000000000]
    R9:  [0x00007F051C129F60]
    R10:  [0x00007F051A800000]
    R11:  [0x00007F0517B37740]
    R12:  [0x0000000000002000]
    R13:  [0x00007F0517806000]
    R14:  [0x00000000000000DE]
    R15:  [0x0000000017B37700]
    EFL:  [0x0000000000010206]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000006]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]

 OS: Linux
 Arch: x86-64

 Backtrace:
  [0x00007F051C0E6BC6] free + 598 (/opt/splunk/lib/libjemalloc.so.1)
  [0x000000000105888D] _ZNSt8_Rb_treeI3StrSt4pairIKS0_16StatsInclOptionsESt10_Select1stIS4_ESt4lessIS0_ESaIS4_EE8_M_eraseEPSt13_Rb_tree_nodeIS4_E + 77 ([splunkd)
  [0x00000000010588F5] _ZN13ScopedPointerISt3mapI3Str16StatsInclOptionsSt4lessIS1_ESaISt4pairIKS1_S2_EEEED2Ev + 21 ([splunkd)
  [0x000000000104753A] _ZNK14StatsProcessor15output_internalEmR13SearchResultsR17SearchResultsInfobbb + 330 ([splunkd)
  [0x0000000001056E14] _ZN14StatsProcessor15reduce_and_emitER13SearchResultsR17SearchResultsInfob + 1956 ([splunkd)
  [0x0000000000FF2A72] _ZN17NewChartProcessor14execute_outputER13SearchResultsR17SearchResultsInfo + 2530 ([splunkd)
  [0x0000000000FFBE55] _ZN17NewChartProcessor31execute_output_and_set_metadataER13SearchResultsR17SearchResultsInfo + 69 ([splunkd)
  [0x0000000000F00DE3] _ZN14DispatchThread15generatePreviewERK8Pathname + 3347 ([splunkd)
  [0x0000000000F013EF] _ZN14DispatchThread15generatePreviewERK8PathnameRK3Str + 47 ([splunkd)
  [0x0000000000F04218] _ZN14DispatchThread8mainImplEv + 11656 ([splunkd)
  [0x0000000000F0F0EF] _ZN14DispatchThread4mainEv + 223 ([splunkd)
  [0x0000000000D81A2D] _ZN6Thread8callMainEPv + 61 ([splunkd)
  [0x00007F051B132E9A] ? (/lib/x86_64-linux-gnu/libpthread.so.0)
  [0x00007F051AE5ECCD] clone + 109 (/lib/x86_64-linux-gnu/libc.so.6)
 Linux / splunk01 / 3.5.0-36-generic / #57-Ubuntu SMP Wed Jun 19 15:10:49 UTC 2013 / x86_64
 /etc/debian_version: wheezy/sid
 MAP: 00400000-017a5000 r-xp 00000000 fc:00 25431840                           /opt/splunk/bin/splunkd
 MAP: 017a5000-017b9000 rw-p 013a5000 fc:00 25431840                           /opt/splunk/bin/splunkd
 MAP: 017b9000-017f4000 rw-p 00000000 00:00 0                                  [heap]
 MAP: 7f05155ea000-7f05155ff000 r-xp 00000000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f05155ff000-7f05157fe000 ---p 00015000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f05157fe000-7f05157ff000 r--p 00014000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f05157ff000-7f0515800000 rw-p 00015000 fc:00 9437447                    /lib/x86_64-linux-gnu/libgcc_s.so.1
 MAP: 7f0515800000-7f0516c00000 rw-p 00000000 00:00 0 
 MAP: 7f0516dfd000-7f0516dfe000 ---p 00000000 00:00 0 
 MAP: 7f0516dfe000-7f0516ffe000 rw-p 00000000 00:00 0                          [stack:22393]
 MAP: 7f0516ffe000-7f0516fff000 ---p 00000000 00:00 0 
 MAP: 7f0516fff000-7f05171ff000 rw-p 00000000 00:00 0 
 MAP: 7f05171ff000-7f0517200000 ---p 00000000 00:00 0 
 MAP: 7f0517200000-7f0517c00000 rw-p 00000000 00:00 0 
 MAP: 7f0517dfe000-7f0517dff000 ---p 00000000 00:00 0 
 MAP: 7f0517dff000-7f0517fff000 rw-p 00000000 00:00 0 
 MAP: 7f0517fff000-7f0518000000 ---p 00000000 00:00 0 
 MAP: 7f0518000000-7f0519000000 rw-p 00000000 00:00 0                          [stack:20998]
 MAP: 7f05191ff000-7f0519200000 ---p 00000000 00:00 0 
 MAP: 7f0519200000-7f0519800000 rw-p 00000000 00:00 0                          [stack:20972]
 MAP: 7f05199fd000-7f05199fe000 ---p 00000000 00:00 0 
 MAP: 7f05199fe000-7f0519bfe000 rw-p 00000000 00:00 0                          [stack:20971]
 MAP: 7f0519bfe000-7f0519bff000 ---p 00000000 00:00 0 
 MAP: 7f0519bff000-7f0519dff000 rw-p 00000000 00:00 0                          [stack:20970]
 MAP: 7f0519dff000-7f0519e00000 ---p 00000000 00:00 0 
 MAP: 7f0519e00000-7f051a000000 rw-p 00000000 00:00 0                          [stack:20968]
 MAP: 7f051a000000-7f051a400000 rw-p 00000000 00:00 0 
 MAP: 7f051a400000-7f051ac00000 rw-p 00000000 00:00 0 
 MAP: 7f051ad5c000-7f051ad5d000 ---p 00000000 00:00 0 
 MAP: 7f051ad5d000-7f051ad66000 rw-p 00000000 00:00 0 
 MAP: 7f051ad66000-7f051ad67000 ---p 00000000 00:00 0 
 MAP: 7f051ad67000-7f051ad6b000 rw-p 00000000 00:00 0 
 MAP: 7f051ad6b000-7f051af20000 r-xp 00000000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051af20000-7f051b11f000 ---p 001b5000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051b11f000-7f051b123000 r--p 001b4000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051b123000-7f051b125000 rw-p 001b8000 fc:00 9440973                    /lib/x86_64-linux-gnu/libc-2.15.so
 MAP: 7f051b125000-7f051b12b000 rw-p 00000000 00:00 0 
 MAP: 7f051b12b000-7f051b143000 r-xp 00000000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b143000-7f051b342000 ---p 00018000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b342000-7f051b343000 r--p 00017000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b343000-7f051b344000 rw-p 00018000 fc:00 9440956                    /lib/x86_64-linux-gnu/libpthread-2.15.so
 MAP: 7f051b344000-7f051b348000 rw-p 00000000 00:00 0 
 MAP: 7f051b348000-7f051b443000 r-xp 00000000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b443000-7f051b642000 ---p 000fb000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b642000-7f051b643000 r--p 000fa000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b643000-7f051b644000 rw-p 000fb000 fc:00 9440969                    /lib/x86_64-linux-gnu/libm-2.15.so
 MAP: 7f051b644000-7f051b65b000 r-xp 00000000 fc:00 25429406                   /opt/splunk/lib/libz.so.1.2.8
 MAP: 7f051b65b000-7f051b65c000 rw-p 00016000 fc:00 25429406                   /opt/splunk/lib/libz.so.1.2.8
 MAP: 7f051b65c000-7f051b6e8000 r-xp 00000000 fc:00 25431661                   /opt/splunk/lib/libsqlite3.so.0.8.6
 MAP: 7f051b6e8000-7f051b6eb000 rw-p 0008b000 fc:00 25431661                   /opt/splunk/lib/libsqlite3.so.0.8.6
 MAP: 7f051b6eb000-7f051b6ed000 r-xp 00000000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b6ed000-7f051b8ed000 ---p 00002000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b8ed000-7f051b8ee000 r--p 00002000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b8ee000-7f051b8ef000 rw-p 00003000 fc:00 9440971                    /lib/x86_64-linux-gnu/libdl-2.15.so
 MAP: 7f051b8ef000-7f051bb18000 r-xp 00000000 fc:00 25429404                   /opt/splunk/lib/libcrypto.so.1.0.0
 MAP: 7f051bb18000-7f051bb42000 rw-p 00228000 fc:00 25429404                   /opt/splunk/lib/libcrypto.so.1.0.0
 MAP: 7f051bb42000-7f051bb46000 rw-p 00000000 00:00 0 
 MAP: 7f051bb46000-7f051bcf7000 r-xp 00000000 fc:00 25431658                   /opt/splunk/lib/libxml2.so.2.9.1
 MAP: 7f051bcf7000-7f051bd01000 rw-p 001b1000 fc:00 25431658                   /opt/splunk/lib/libxml2.so.2.9.1
 MAP: 7f051bd01000-7f051bd02000 rw-p 00000000 00:00 0 
 MAP: 7f051bd02000-7f051bd09000 r-xp 00000000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bd09000-7f051bf08000 ---p 00007000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bf08000-7f051bf09000 r--p 00006000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bf09000-7f051bf0a000 rw-p 00007000 fc:00 9440960                    /lib/x86_64-linux-gnu/librt-2.15.so
 MAP: 7f051bf0a000-7f051bf2c000 r-xp 00000000 fc:00 9440961                    /lib/x86_64-linux-gnu/ld-2.15.so
 MAP: 7f051bf36000-7f051bf37000 rw-p 00000000 00:00 0 
 MAP: 7f051bf37000-7f051bf46000 r-xp 00000000 fc:00 25429410                   /opt/splunk/lib/libbz2.so.1.0.3
 MAP: 7f051bf46000-7f051bf48000 rw-p 0000e000 fc:00 25429410                   /opt/splunk/lib/libbz2.so.1.0.3
 MAP: 7f051bf48000-7f051bfd7000 r-xp 00000000 fc:00 25429401                   /opt/splunk/lib/libarchive.so.13.1.2
 MAP: 7f051bfd7000-7f051bfdb000 rw-p 0008e000 fc:00 25429401                   /opt/splunk/lib/libarchive.so.13.1.2
 MAP: 7f051bfdb000-7f051bfdd000 rw-p 00000000 00:00 0 
 MAP: 7f051bfdd000-7f051c03e000 r-xp 00000000 fc:00 25431660                   /opt/splunk/lib/libssl.so.1.0.0
 MAP: 7f051c03e000-7f051c048000 rw-p 00061000 fc:00 25431660                   /opt/splunk/lib/libssl.so.1.0.0
 MAP: 7f051c048000-7f051c049000 rw-p 00000000 00:00 0 
 MAP: 7f051c049000-7f051c097000 r-xp 00000000 fc:00 25431659                   /opt/splunk/lib/libxslt.so.1.1.28
 MAP: 7f051c097000-7f051c099000 rw-p 0004d000 fc:00 25431659                   /opt/splunk/lib/libxslt.so.1.1.28
 MAP: 7f051c099000-7f051c0d9000 r-xp 00000000 fc:00 25431644                   /opt/splunk/lib/libpcre.so.1.2.1
 MAP: 7f051c0d9000-7f051c0da000 rw-p 0003f000 fc:00 25431644                   /opt/splunk/lib/libpcre.so.1.2.1
 MAP: 7f051c0da000-7f051c0db000 rw-p 00000000 00:00 0 
 MAP: 7f051c0e2000-7f051c0e3000 rw-p 00000000 00:00 0 
 MAP: 7f051c0e3000-7f051c116000 r-xp 00000000 fc:00 25431801                   /opt/splunk/lib/libjemalloc.so.1
 MAP: 7f051c116000-7f051c118000 rw-p 00033000 fc:00 25431801                   /opt/splunk/lib/libjemalloc.so.1
 MAP: 7f051c118000-7f051c12c000 rw-p 00000000 00:00 0 
 MAP: 7f051c12c000-7f051c12d000 r--p 00022000 fc:00 9440961                    /lib/x86_64-linux-gnu/ld-2.15.so
 MAP: 7f051c12d000-7f051c12f000 rw-p 00023000 fc:00 9440961                    /lib/x86_64-linux-gnu/ld-2.15.so
 MAP: 7fff53226000-7fff53247000 rw-p 00000000 00:00 0                          [stack]
 MAP: 7fff5325f000-7fff53260000 r-xp 00000000 00:00 0                          [vdso]
 MAP: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Last errno: 0
Threads running: 7
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=29117] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=29117] search --id=1396993215.369 --maxbuckets=0 --ttl=600 --maxout=50000000 --maxtime=8640000 --lookups=1 --reduce_freq=10 --user=mdegan --pro --roles=admin:can_delete:power:user
Thread: "dispatch", did_join=1, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f05185a0010:
00000000  00 f7 3f 18 05 7f 00 00                           |..?.....|
00000008

x86 CPUID registers:
         0: 0000000B 756E6547 6C65746E 49656E69
         1: 00020651 03080800 82982203 1FEBFBFF
         2: 55035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00001000 00001000 00000003 00001120
         6: 00000007 00000001 00000009 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000001 00000000 00000000 00000000
         A: 07300403 0000007F 00000000 00000000
         B: 00000000 00000000 1B139CCD 00000003
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 6E6F6558 20295228 20555043 202D3745
  80000004: 30373832 20402020 30342E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003028 00000000 00000000 00000000
terminating...
Tags (2)

mrgibbon
Contributor

I managed to work around this by un-taring the current version of Splunk over the top of the installation.
Running a chown command to make sure the files were all owned by the right user, then starting up again.
Worked for me, hope this can help someone else.

0 Karma

kamal_jagga
Contributor

Would you be able to advise the directories/files whose permissions you changed. This happened on 3 out of 8 of my indexers with a fraction of minute.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...