Why does Splunk Enterprise 7.1.2 crashes sporadically every few days?

[scc00]

HI My Splunk Enterprise 7.1.2 keeps crashing sporadically, every few days. See crash log below.

[build a0c72a66db66] 2018-08-01 18:26:40
Received fatal signal 11 (Segmentation fault).
Cause:
No memory mapped at address [0x0000000000000010].
Crashing thread: TcpChannelThread
Registers:
RIP: [0x0000556D51DB0820] ZN16SearchResultsMem23appendSharedDestructiveERS + 208 (splunkd + 0x1029820)
RDI: [0x0000000000000000]
RSI: [0x00007F2618DA5E76]
RBP: [0x00000000000000F8]
RSP: [0x00007F25C51FCB30]
RAX: [0x0000000000000000]
RBX: [0x00007F25AB1D3F00]
RCX: [0x0000000000000000]
RDX: [0x0000000000000006]
R8: [0x0000000000000000]
R9: [0x00007F266BB50040]
R10: [0x00007F266DA027D0]
R11: [0x00007F266C4CBAA0]
R12: [0x00007F25AB1D3F00]
R13: [0x00007F25C51FCC20]
R14: [0x00007F2660ABFC28]
R15: [0x00007F25C51FCB80]
EFL: [0x0000000000010283]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000006]
CSGSFS: [0x002B000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x0000556D51DB0820] ZN16SearchResultsMem23appendSharedDestructiveERS + 208 (splunkd + 0x1029820)
[0x0000556D528050B2] _ZN18GeoFilterProcessor16filterUsingQtreeER18SearchResultsFilesP8QuadTree + 530 (splunkd + 0x1A7E0B2)
[0x0000556D5280637E] _ZN18GeoFilterProcessor7executeER18SearchResultsFilesR17SearchResultsInfo + 670 (splunkd + 0x1A7F37E)
[0x0000556D525E66DD] _ZN15SearchProcessor16execute_dispatchER18SearchResultsFilesR17SearchResultsInfoRK3Str + 237 (splunkd + 0x185F6DD)
[0x0000556D525D6587] _ZN14SearchPipeline7executeER18SearchResultsFilesR17SearchResultsInfo + 279 (splunkd + 0x184F587)
[0x0000556D529ED18A] _ZN22HandleJobsDataProvider19handleResultsGetterEPN14DispatchSearch28GenericOutputResultsAcceptorERK3StrRK18SearchResultsFilesRK10StrSegmentbRK8Pathname + 3914 (splunkd + 0x1C6618A)
[0x0000556D529EFA20] _ZN22HandleJobsDataProvider31executeEventResultPreviewActionERK3StrRK19SearchJobStatusDataRK13HttpArgumentsP6StrSetRK10StrSegmentb + 144 (splunkd + 0x1C68A20)
[0x0000556D529EFB87] _ZN22HandleJobsDataProvider17handleStatusQueryERK21UserTimezoneSpecifierP6StrSetRK3StrP24CachedJobStatusReferenceRK10StrSegmentRK10HttpMethodRK13HttpArgumentsRb + 215 (splunkd + 0x1C68B87)
[0x0000556D529F2366] _ZN22HandleJobsDataProvider10handleJobsERK21UserTimezoneSpecifier10HttpMethodRK3StrRK10StrSegmentbRK13HttpArguments + 1094 (splunkd + 0x1C6B366)
[0x0000556D529F3482] _ZN22HandleJobsDataProvider18handleWithTimezoneERK21UserTimezoneSpecifier + 754 (splunkd + 0x1C6C482)
[0x0000556D52A20B23] _ZN38DispatchSearchDataProviderWithTimezone21handleWithoutTimezoneEv + 435 (splunkd + 0x1C99B23)
[0x0000556D529ABB39] _ZN26DispatchSearchDataProvider2goEv + 41 (splunkd + 0x1C24B39)
[0x0000556D51C67348] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0xEE0348)
[0x0000556D51C5C70F] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 31 (splunkd + 0xED570F)
[0x0000556D5210AF20] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 272 (splunkd + 0x1383F20)
[0x0000556D5163138A] _ZN16TcpChannelThread4mainEv + 218 (splunkd + 0x8AA38A)
[0x0000556D5219794F] _ZN6Thread8callMainEPv + 111 (splunkd + 0x141094F)
[0x00007F266C713DE5] ? (libpthread.so.0 + 0x7DE5)
[0x00007F266C43DBAD] clone + 109 (libc.so.6 + 0xFEBAD)
Linux / ip-172-31-10-74 / 4.14.47-56.37.amzn1.x86_64 / #1 SMP Wed Jun 6 18:49:01 UTC 2018 / x86_64
/etc/os-release: NAME="Amazon Linux AMI"
glibc version: 2.17
glibc release: stable
MAP: 556d50d87000-556d53543000 r-xp 00000000 103:01 394751 /opt/splunk/bin/splunkd
MAP: 556d53543000-556d535fb000 rw-p 027bc000 103:01 394751 /opt/splunk/bin/splunkd
MAP: 556d535fb000-556d5367a000 rw-p 00000000 00:00 0
MAP: 7f2582200000-7f25bd600000 rw-p 00000000 00:00 0
MAP: 7f25bd7ff000-7f25bd800000 ---p 00000000 00:00 0
MAP: 7f25bd800000-7f25bdc00000 rw-p 00000000 00:00 0
MAP: 7f25bdc00000-7f25be800000 rw-p 00000000 00:00 0
MAP: 7f25be9ff000-7f25bea00000 ---p 00000000 00:00 0
MAP: 7f25bea00000-7f25bec00000 rw-p 00000000 00:00 0
MAP: 7f25bec00000-7f25bfc00000 rw-p 00000000 00:00 0
MAP: 7f25bfe00000-7f25c0000000 rw-p 00000000 00:00 0
MAP: 7f25c01fd000-7f25c01fe000 ---p 00000000 00:00 0
MAP: 7f25c01fe000-7f25c03fe000 rw-p 00000000 00:00 0
MAP: 7f25c03fe000-7f25c03ff000 ---p 00000000 00:00 0
MAP: 7f25c03ff000-7f25c05ff000 rw-p 00000000 00:00 0
MAP: 7f25c05ff000-7f25c0600000 ---p 00000000 00:00 0
MAP: 7f25c0600000-7f25c0800000 rw-p 00000000 00:00 0
MAP: 7f25c0800000-7f25c2600000 rw-p 00000000 00:00 0
MAP: 7f25c27fe000-7f25c27ff000 ---p 00000000 00:00 0
MAP: 7f25c27ff000-7f25c2bff000 rw-p 00000000 00:00 0
MAP: 7f25c2dff000-7f25c2e00000 ---p 00000000 00:00 0
MAP: 7f25c2e00000-7f25c3000000 rw-p 00000000 00:00 0
MAP: 7f25c3000000-7f25c4e00000 rw-p 00000000 00:00 0
MAP: 7f25c4fff000-7f25c5000000 ---p 00000000 00:00 0
MAP: 7f25c5000000-7f25c5200000 rw-p 00000000 00:00 0
MAP: 7f25c5200000-7f25c6000000 rw-p 00000000 00:00 0
MAP: 7f25c61ff000-7f25c6200000 ---p 00000000 00:00 0
MAP: 7f25c6200000-7f25c6400000 rw-p 00000000 00:00 0
MAP: 7f25c6400000-7f25c6600000 rw-p 00000000 00:00 0
MAP: 7f25c67fe000-7f25c67ff000 ---p 00000000 00:00 0
MAP: 7f25c67ff000-7f25c69ff000 rw-p 00000000 00:00 0
MAP: 7f25c69ff000-7f25c6a00000 ---p 00000000 00:00 0
MAP: 7f25c6a00000-7f25c6c00000 rw-p 00000000 00:00 0
MAP: 7f25c6c00000-7f25c7800000 rw-p 00000000 00:00 0
MAP: 7f25c7800000-7f25c8c00000 rw-p 00000000 00:00 0
MAP: 7f25c8c00000-7f25c9000000 rw-p 00000000 00:00 0
MAP: 7f25c9200000-7f25c9600000 rw-p 00000000 00:00 0
MAP: 7f25c97e7000-7f25c97fd000 r-xp 00000000 103:01 2390 /lib64/libgcc_s-7-20170915.so.1
MAP: 7f25c97fd000-7f25c99fc000 ---p 00016000 103:01 2390 /lib64/libgcc_s-7-20170915.so.1
MAP: 7f25c99fc000-7f25c99fd000 rw-p 00015000 103:01 2390 /lib64/libgcc_s-7-20170915.so.1
MAP: 7f25c99fd000-7f25c99fe000 ---p 00000000 00:00 0
MAP: 7f25c99fe000-7f25c9bfe000 rw-p 00000000 00:00 0
MAP: 7f25c9dff000-7f25c9e00000 ---p 00000000 00:00 0
MAP: 7f25c9e00000-7f25ca000000 rw-p 00000000 00:00 0
MAP: 7f25ca000000-7f25ca600000 rw-p 00000000 00:00 0
MAP: 7f25ca7ff000-7f25ca800000 ---p 00000000 00:00 0
MAP: 7f25ca800000-7f25cac00000 rw-p 00000000 00:00 0
MAP: 7f25cac00000-7f25cb400000 rw-p 00000000 00:00 0
MAP: 7f25cb400000-7f25cd600000 rw-p 00000000 00:00 0
MAP: 7f25cd600000-7f25cee00000 rw-p 00000000 00:00 0
MAP: 7f25ceffe000-7f25cefff000 ---p 00000000 00:00 0
MAP: 7f25cefff000-7f25cf3ff000 rw-p 00000000 00:00 0
MAP: 7f25cf3ff000-7f25cf400000 ---p 00000000 00:00 0
MAP: 7f25cf400000-7f25cf800000 rw-p 00000000 00:00 0
MAP: 7f25cf800000-7f25d0000000 rw-p 00000000 00:00 0
MAP: 7f25d0000000-7f25d1400000 rw-p 00000000 00:00 0
MAP: 7f25d1400000-7f25d1c00000 rw-p 00000000 00:00 0
MAP: 7f25d1c00000-7f25d2000000 rw-p 00000000 00:00 0
MAP: 7f25d21fd000-7f25d21fe000 ---p 00000000 00:00 0
MAP: 7f25d21fe000-7f25d25fe000 rw-p 00000000 00:00 0
MAP: 7f25d25fe000-7f25d25ff000 ---p 00000000 00:00 0
MAP: 7f25d25ff000-7f25d27ff000 rw-p 00000000 00:00 0
MAP: 7f25d27ff000-7f25d2800000 ---p 00000000 00:00 0
MAP: 7f25d2800000-7f25d2a00000 rw-p 00000000 00:00 0
MAP: 7f25d2a00000-7f25d5600000 rw-p 00000000 00:00 0
MAP: 7f25d5600000-7f25d7800000 rw-p 00000000 00:00 0
MAP: 7f25d7800000-7f25d8600000 rw-p 00000000 00:00 0
MAP: 7f25d8600000-7f25d9e00000 rw-p 00000000 00:00 0
MAP: 7f25d9e00000-7f25dc800000 rw-p 00000000 00:00 0
MAP: 7f25dc800000-7f25dd800000 rw-p 00000000 00:00 0
MAP: 7f25dd800000-7f25de000000 rw-p 00000000 00:00 0
MAP: 7f25de000000-7f25de800000 rw-p 00000000 00:00 0
MAP: 7f25de800000-7f25dee00000 rw-p 00000000 00:00 0
MAP: 7f25dee00000-7f25e0200000 rw-p 00000000 00:00 0
MAP: 7f25e0200000-7f25e1000000 rw-p 00000000 00:00 0
MAP: 7f25e1000000-7f25e2400000 rw-p 00000000 00:00 0
MAP: 7f25e2400000-7f25e3800000 rw-p 00000000 00:00 0
MAP: 7f25e3800000-7f25e3c00000 rw-p 00000000 00:00 0
MAP: 7f25e3c00000-7f25e4c00000 rw-p 00000000 00:00 0
MAP: 7f25e4c00000-7f25e5000000 rw-p 00000000 00:00 0
MAP: 7f25e5000000-7f25e8400000 rw-p 00000000 00:00 0
MAP: 7f25e8400000-7f25e8c00000 rw-p 00000000 00:00 0
MAP: 7f25e8c00000-7f25e9600000 rw-p 00000000 00:00 0
MAP: 7f25e9600000-7f25ea400000 rw-p 00000000 00:00 0
MAP: 7f25ea400000-7f25eac00000 rw-p 00000000 00:00 0
MAP: 7f25eac00000-7f25ec200000 rw-p 00000000 00:00 0
MAP: 7f25ec200000-7f25ed800000 rw-p 00000000 00:00 0
MAP: 7f25ed800000-7f25ee800000 rw-p 00000000 00:00 0
MAP: 7f25ee800000-7f25ef400000 rw-p 00000000 00:00 0
MAP: 7f25ef400000-7f25f0000000 rw-p 00000000 00:00 0
MAP: 7f25f0000000-7f25f1400000 rw-p 00000000 00:00 0
MAP: 7f25f1400000-7f25f2000000 rw-p 00000000 00:00 0
MAP: 7f25f2000000-7f25f4e00000 rw-p 00000000 00:00 0
MAP: 7f25f4e00000-7f25f6200000 rw-p 00000000 00:00 0
MAP: 7f25f6200000-7f25f8200000 rw-p 00000000 00:00 0
MAP: 7f25f8200000-7f25f8e00000 rw-p 00000000 00:00 0
MAP: 7f25f8e00000-7f25fb800000 rw-p 00000000 00:00 0
MAP: 7f25fb800000-7f25fea00000 rw-p 00000000 00:00 0
MAP: 7f25fea00000-7f260be00000 rw-p 00000000 00:00 0
MAP: 7f260be00000-7f260ca00000 rw-p 00000000 00:00 0
MAP: 7f260ca00000-7f260d200000 rw-p 00000000 00:00 0
MAP: 7f260d200000-7f260d800000 rw-p 00000000 00:00 0
MAP: 7f260d800000-7f260de00000 rw-p 00000000 00:00 0
MAP: 7f260de00000-7f260e400000 rw-p 00000000 00:00 0
MAP: 7f260e400000-7f260e800000 rw-p 00000000 00:00 0
MAP: 7f260e800000-7f260f200000 rw-p 00000000 00:00 0
MAP: 7f260f200000-7f260fe00000 rw-p 00000000 00:00 0
MAP: 7f260fe00000-7f2610400000 rw-p 00000000 00:00 0
MAP: 7f2610400000-7f2610a00000 rw-p 00000000 00:00 0
MAP: 7f2610a00000-7f2611000000 rw-p 00000000 00:00 0
MAP: 7f2611000000-7f2615a00000 rw-p 00000000 00:00 0
MAP: 7f2615bff000-7f2615c00000 ---p 00000000 00:00 0
MAP: 7f2615c00000-7f2615e00000 rw-p 00000000 00:00 0
MAP: 7f2615e00000-7f2616600000 rw-p 00000000 00:00 0
MAP: 7f2616600000-7f2618200000 rw-p 00000000 00:00 0
MAP: 7f26183ff000-7f2618400000 ---p 00000000 00:00 0
MAP: 7f2618400000-7f2618600000 rw-p 00000000 00:00 0
MAP: 7f2618600000-7f2618e00000 rw-p 00000000 00:00 0
MAP: 7f2618e00000-7f261a200000 rw-p 00000000 00:00 0
MAP: 7f261a200000-7f261a800000 rw-p 00000000 00:00 0
MAP: 7f261a800000-7f261b200000 rw-p 00000000 00:00 0
MAP: 7f261b200000-7f261ba00000 rw-p 00000000 00:00 0
MAP: 7f261ba00000-7f261c000000 rw-p 00000000 00:00 0
MAP: 7f261c000000-7f261c800000 rw-p 00000000 00:00 0
MAP: 7f261c800000-7f2620600000 rw-p 00000000 00:00 0
MAP: 7f2620600000-7f2621800000 rw-p 00000000 00:00 0
MAP: 7f2621800000-7f2625800000 rw-p 00000000 00:00 0
MAP: 7f2625800000-7f2626200000 rw-p 00000000 00:00 0
MAP: 7f2626200000-7f2626a00000 rw-p 00000000 00:00 0
MAP: 7f2626a00000-7f2627600000 rw-p 00000000 00:00 0
MAP: 7f2627600000-7f2628200000 rw-p 00000000 00:00 0
MAP: 7f2628200000-7f2628600000 rw-p 00000000 00:00 0
MAP: 7f2628600000-7f262b200000 rw-p 00000000 00:00 0
MAP: 7f262b200000-7f262d400000 rw-p 00000000 00:00 0
MAP: 7f262d400000-7f2630a00000 rw-p 00000000 00:00 0
MAP: 7f2630a00000-7f2631600000 rw-p 00000000 00:00 0
MAP: 7f26317ff000-7f2631800000 ---p 00000000 00:00 0
MAP: 7f2631800000-7f2631a00000 rw-p 00000000 00:00 0
MAP: 7f2631a00000-7f2635200000 rw-p 00000000 00:00 0
MAP: 7f26353cc000-7f26353e2000 r-xp 00000000 103:01 2485 /lib64/libresolv-2.17.so
MAP: 7f26353e2000-7f26355e1000 ---p 00016000 103:01 2485 /lib64/libresolv-2.17.so
MAP: 7f26355e1000-7f26355e2000 r--p 00015000 103:01 2485 /lib64/libresolv-2.17.so
MAP: 7f26355e2000-7f26355e3000 rw-p 00016000 103:01 2485 /lib64/libresolv-2.17.so
MAP: 7f26355e3000-7f26355e5000 rw-p 00000000 00:00 0
MAP: 7f26355e5000-7f26355ea000 r-xp 00000000 103:01 2473 /lib64/libnss_dns-2.17.so
MAP: 7f26355ea000-7f26357ea000 ---p 00005000 103:01 2473 /lib64/libnss_dns-2.17.so
MAP: 7f26357ea000-7f26357eb000 r--p 00005000 103:01 2473 /lib64/libnss_dns-2.17.so
MAP: 7f26357eb000-7f26357ec000 rw-p 00006000 103:01 2473 /lib64/libnss_dns-2.17.so
MAP: 7f26357ec000-7f26357f8000 r-xp 00000000 103:01 2475 /lib64/libnss_files-2.17.so
MAP: 7f26357f8000-7f26359f7000 ---p 0000c000 103:01 2475 /lib64/libnss_files-2.17.so
MAP: 7f26359f7000-7f26359f8000 r--p 0000b000 103:01 2475 /lib64/libnss_files-2.17.so
MAP: 7f26359f8000-7f26359f9000 rw-p 0000c000 103:01 2475 /lib64/libnss_files-2.17.so
MAP: 7f26359f9000-7f26359ff000 rw-p 00000000 00:00 0
MAP: 7f2635a00000-7f2638800000 rw-p 00000000 00:00 0
MAP: 7f2638800000-7f2638e00000 rw-p 00000000 00:00 0
MAP: 7f2638e00000-7f263b000000 rw-p 00000000 00:00 0
MAP: 7f263b1ff000-7f263b200000 ---p 00000000 00:00 0
MAP: 7f263b200000-7f263b400000 rw-p 00000000 00:00 0
MAP: 7f263b400000-7f2644200000 rw-p 00000000 00:00 0
MAP: 7f2644200000-7f2644600000 rw-p 00000000 00:00 0
MAP: 7f26447ff000-7f2644800000 ---p 00000000 00:00 0
MAP: 7f2644800000-7f2644a00000 rw-p 00000000 00:00 0
MAP: 7f2644a00000-7f2646400000 rw-p 00000000 00:00 0
MAP: 7f26465ff000-7f2646600000 ---p 00000000 00:00 0
MAP: 7f2646600000-7f2646e00000 rw-p 00000000 00:00 0
MAP: 7f2646e00000-7f2647000000 rw-p 00000000 00:00 0
MAP: 7f26471ff000-7f2647200000 ---p 00000000 00:00 0
MAP: 7f2647200000-7f2647400000 rw-p 00000000 00:00 0
MAP: 7f2647400000-7f2647600000 rw-p 00000000 00:00 0
MAP: 7f26477ff000-7f2647800000 ---p 00000000 00:00 0
MAP: 7f2647800000-7f2648000000 rw-p 00000000 00:00 0
MAP: 7f2648000000-7f2649e00000 rw-p 00000000 00:00 0
MAP: 7f2649e00000-7f264ca00000 rw-p 00000000 00:00 0
MAP: 7f264ca00000-7f264e800000 rw-p 00000000 00:00 0
MAP: 7f264e9ff000-7f264ea00000 ---p 00000000 00:00 0
MAP: 7f264ea00000-7f264ec00000 rw-p 00000000 00:00 0
MAP: 7f264ec00000-7f264f400000 rw-p 00000000 00:00 0
MAP: 7f264f5ff000-7f264f600000 ---p 00000000 00:00 0
MAP: 7f264f600000-7f264f800000 rw-p 00000000 00:00 0
MAP: 7f264f800000-7f2650000000 rw-p 00000000 00:00 0
MAP: 7f26501ff000-7f2650200000 ---p 00000000 00:00 0
MAP: 7f2650200000-7f2650400000 rw-p 00000000 00:00 0
MAP: 7f2650400000-7f2650600000 rw-p 00000000 00:00 0
MAP: 7f26507ff000-7f2650800000 ---p 00000000 00:00 0
MAP: 7f2650800000-7f2650a00000 rw-p 00000000 00:00 0
MAP: 7f2650a00000-7f2651c00000 rw-p 00000000 00:00 0
MAP: 7f2651dff000-7f2651e00000 ---p 00000000 00:00 0
MAP: 7f2651e00000-7f2652000000 rw-p 00000000 00:00 0
MAP: 7f2652000000-7f2652200000 rw-p 00000000 00:00 0
MAP: 7f26523ff000-7f2652400000 ---p 00000000 00:00 0
MAP: 7f2652400000-7f2652600000 rw-p 00000000 00:00 0
MAP: 7f2652600000-7f2652800000 rw-p 00000000 00:00 0
MAP: 7f26529ff000-7f2652a00000 ---p 00000000 00:00 0
MAP: 7f2652a00000-7f2652c00000 rw-p 00000000 00:00 0
MAP: 7f2652c00000-7f2652e00000 rw-p 00000000 00:00 0
MAP: 7f2652fff000-7f2653000000 ---p 00000000 00:00 0
MAP: 7f2653000000-7f2653200000 rw-p 00000000 00:00 0
MAP: 7f2653200000-7f2653400000 rw-p 00000000 00:00 0
MAP: 7f26535ff000-7f2653600000 ---p 00000000 00:00 0
MAP: 7f2653600000-7f2653800000 rw-p 00000000 00:00 0
MAP: 7f2653800000-7f2653a00000 rw-p 00000000 00:00 0
MAP: 7f2653bfe000-7f2653bff000 ---p 00000000 00:00 0
MAP: 7f2653bff000-7f2653dff000 rw-p 00000000 00:00 0
MAP: 7f2653dff000-7f2653e00000 ---p 00000000 00:00 0
MAP: 7f2653e00000-7f2654000000 rw-p 00000000 00:00 0
MAP: 7f2654000000-7f2654200000 rw-p 00000000 00:00 0
MAP: 7f26543ff000-7f2654400000 ---p 00000000 00:00 0
MAP: 7f2654400000-7f2654600000 rw-p 00000000 00:00 0
MAP: 7f2654600000-7f2654800000 rw-p 00000000 00:00 0
MAP: 7f26549ff000-7f2654a00000 ---p 00000000 00:00 0
MAP: 7f2654a00000-7f2654c00000 rw-p 00000000 00:00 0
MAP: 7f2654c00000-7f2655400000 rw-p 00000000 00:00 0
MAP: 7f26555ff000-7f2655600000 ---p 00000000 00:00 0
MAP: 7f2655600000-7f2655800000 rw-p 00000000 00:00 0
MAP: 7f2655800000-7f2656600000 rw-p 00000000 00:00 0
MAP: 7f26567ff000-7f2656800000 ---p 00000000 00:00 0
MAP: 7f2656800000-7f2656a00000 rw-p 00000000 00:00 0
MAP: 7f2656a00000-7f2656c00000 rw-p 00000000 00:00 0
MAP: 7f2656dff000-7f2656e00000 ---p 00000000 00:00 0
MAP: 7f2656e00000-7f2657000000 rw-p 00000000 00:00 0
MAP: 7f2657000000-7f2658e00000 rw-p 00000000 00:00 0
MAP: 7f2658fff000-7f2659000000 ---p 00000000 00:00 0
MAP: 7f2659000000-7f2659200000 rw-p 00000000 00:00 0
MAP: 7f2659200000-7f2659400000 rw-p 00000000 00:00 0
MAP: 7f26595ff000-7f2659600000 ---p 00000000 00:00 0
MAP: 7f2659600000-7f2659800000 rw-p 00000000 00:00 0
MAP: 7f2659800000-7f2659a00000 rw-p 00000000 00:00 0
MAP: 7f2659bff000-7f2659c00000 ---p 00000000 00:00 0
MAP: 7f2659c00000-7f2659e00000 rw-p 00000000 00:00 0
MAP: 7f2659e00000-7f265a000000 rw-p 00000000 00:00 0
MAP: 7f265a1ff000-7f265a200000 ---p 00000000 00:00 0
MAP: 7f265a200000-7f265a400000 rw-p 00000000 00:00 0
MAP: 7f265a400000-7f265a600000 rw-p 00000000 00:00 0
MAP: 7f265a7ff000-7f265a800000 ---p 00000000 00:00 0
MAP: 7f265a800000-7f265aa00000 rw-p 00000000 00:00 0
MAP: 7f265aa00000-7f265ac00000 rw-p 00000000 00:00 0
MAP: 7f265adff000-7f265ae00000 ---p 00000000 00:00 0
MAP: 7f265ae00000-7f265b000000 rw-p 00000000 00:00 0
MAP: 7f265b000000-7f265b400000 rw-p 00000000 00:00 0
MAP: 7f265b400000-7f265b600000 rw-p 00000000 00:00 0
MAP: 7f265b7fe000-7f265b7ff000 ---p 00000000 00:00 0
MAP: 7f265b7ff000-7f265b9ff000 rw-p 00000000 00:00 0
MAP: 7f265b9ff000-7f265ba00000 ---p 00000000 00:00 0
MAP: 7f265ba00000-7f265bc00000 rw-p 00000000 00:00 0
MAP: 7f265bc00000-7f265be00000 rw-p 00000000 00:00 0
MAP: 7f265bfff000-7f265c000000 ---p 00000000 00:00 0
MAP: 7f265c000000-7f265c200000 rw-p 00000000 00:00 0
MAP: 7f265c200000-7f265c400000 rw-p 00000000 00:00 0
MAP: 7f265c5ff000-7f265c600000 ---p 00000000 00:00 0
MAP: 7f265c600000-7f265c800000 rw-p 00000000 00:00 0
MAP: 7f265c800000-7f265cc00000 rw-p 00000000 00:00 0
MAP: 7f265cdff000-7f265ce00000 ---p 00000000 00:00 0
MAP: 7f265ce00000-7f265d000000 rw-p 00000000 00:00 0
MAP: 7f265d000000-7f265d200000 rw-p 00000000 00:00 0
MAP: 7f265d3ff000-7f265d400000 ---p 00000000 00:00 0
MAP: 7f265d400000-7f265d600000 rw-p 00000000 00:00 0
MAP: 7f265d600000-7f265d800000 rw-p 00000000 00:00 0
MAP: 7f265d9ff000-7f265da00000 ---p 00000000 00:00 0
MAP: 7f265da00000-7f265dc00000 rw-p 00000000 00:00 0
MAP: 7f265dc00000-7f265f400000 rw-p 00000000 00:00 0
MAP: 7f265f5fe000-7f265f5ff000 ---p 00000000 00:00 0
MAP: 7f265f5ff000-7f265f7ff000 rw-p 00000000 00:00 0
MAP: 7f265f7ff000-7f265f800000 ---p 00000000 00:00 0
MAP: 7f265f800000-7f265fa00000 rw-p 00000000 00:00 0
MAP: 7f265fa00000-7f265fc00000 rw-p 00000000 00:00 0
MAP: 7f265fdff000-7f265fe00000 ---p 00000000 00:00 0
MAP: 7f265fe00000-7f2660000000 rw-p 00000000 00:00 0
MAP: 7f2660000000-7f2660600000 rw-p 00000000 00:00 0
MAP: 7f26607ff000-7f2660800000 ---p 00000000 00:00 0
MAP: 7f2660800000-7f2660a00000 rw-p 00000000 00:00 0
MAP: 7f2660a00000-7f2661000000 rw-p 00000000 00:00 0
MAP: 7f266111e000-7f266115e000 rwxp 00000000 00:00 0
MAP: 7f26611fe000-7f26611ff000 ---p 00000000 00:00 0
MAP: 7f26611ff000-7f26613ff000 rw-p 00000000 00:00 0
MAP: 7f2661400000-7f2661600000 rw-p 00000000 00:00 0
MAP: 7f266169e000-7f266170e000 rwxp 00000000 00:00 0
MAP: 7f26617fe000-7f26617ff000 ---p 00000000 00:00 0
MAP: 7f26617ff000-7f26619ff000 rw-p 00000000 00:00 0
MAP: 7f26619ff000-7f2661a00000 ---p 00000000 00:00 0
MAP: 7f2661a00000-7f2661c00000 rw-p 00000000 00:00 0
MAP: 7f2661c00000-7f2661e00000 rw-p 00000000 00:00 0
MAP: 7f2661e3f000-7f2661ebf000 rwxp 00000000 00:00 0
MAP: 7f2661f2f000-7f2661fcf000 rwxp 00000000 00:00 0
MAP: 7f2661fff000-7f2662000000 ---p 00000000 00:00 0
MAP: 7f2662000000-7f2662200000 rw-p 00000000 00:00 0
MAP: 7f2662200000-7f2662800000 rw-p 00000000 00:00 0
MAP: 7f26628ac000-7f266298c000 rwxp 00000000 00:00 0
MAP: 7f26629fc000-7f26629fd000 ---p 00000000 00:00 0
MAP: 7f26629fd000-7f2662bfd000 rw-p 00000000 00:00 0
MAP: 7f2662c9d000-7f2662d8d000 rwxp 00000000 00:00 0
MAP: 7f2662dfd000-7f2662dfe000 ---p 00000000 00:00 0
MAP: 7f2662dfe000-7f2662ffe000 rw-p 00000000 00:00 0
MAP: 7f2662ffe000-7f2662fff000 ---p 00000000 00:00 0
MAP: 7f2662fff000-7f26631ff000 rw-p 00000000 00:00 0
MAP: 7f26631ff000-7f2663200000 ---p 00000000 00:00 0
MAP: 7f2663200000-7f2663400000 rw-p 00000000 00:00 0
MAP: 7f2663400000-7f2663600000 rw-p 00000000 00:00 0
MAP: 7f266364f000-7f26636cf000 rwxp 00000000 00:00 0
MAP: 7f26636ef000-7f266370f000 rwxp 00000000 00:00 0
MAP: 7f266374f000-7f266376f000 rwxp 00000000 00:00 0
MAP: 7f266379f000-7f26637bf000 rwxp 00000000 00:00 0
MAP: 7f26637ff000-7f2663800000 ---p 00000000 00:00 0
MAP: 7f2663800000-7f2663a00000 rw-p 00000000 00:00 0
MAP: 7f2663a00000-7f2663c00000 rw-p 00000000 00:00 0
MAP: 7f2663c0e000-7f2663c2e000 rwxp 00000000 00:00 0
MAP: 7f2663c9e000-7f2663cae000 rwxp 00000000 00:00 0
MAP: 7f2663cce000-7f2663cde000 rwxp 00000000 00:00 0
MAP: 7f2663cfe000-7f2663d0e000 rwxp 00000000 00:00 0
MAP: 7f2663dfe000-7f2663dff000 ---p 00000000 00:00 0
MAP: 7f2663dff000-7f2663fff000 rw-p 00000000 00:00 0
MAP: 7f2664000000-7f2664600000 rw-p 00000000 00:00 0
MAP: 7f266460e000-7f266461e000 rwxp 00000000 00:00 0
MAP: 7f266464e000-7f266465e000 rwxp 00000000 00:00 0
MAP: 7f26646ce000-7f26646de000 rwxp 00000000 00:00 0
MAP: 7f266472e000-7f266473e000 rwxp 00000000 00:00 0
MAP: 7f266477e000-7f266479e000 rwxp 00000000 00:00 0
MAP: 7f26647fe000-7f26647ff000 ---p 00000000 00:00 0
MAP: 7f26647ff000-7f26649ff000 rw-p 00000000 00:00 0
MAP: 7f26649ff000-7f2664a00000 ---p 00000000 00:00 0
MAP: 7f2664a00000-7f2664c00000 rw-p 00000000 00:00 0
MAP: 7f2664c00000-7f2664e00000 rw-p 00000000 00:00 0
MAP: 7f2664e9f000-7f2664eaf000 rwxp 00000000 00:00 0
MAP: 7f2664f4f000-7f2664f6f000 rwxp 00000000 00:00 0
MAP: 7f2664fcf000-7f2664fef000 rwxp 00000000 00:00 0
MAP: 7f2664fff000-7f2665000000 ---p 00000000 00:00 0
MAP: 7f2665000000-7f2665200000 rw-p 00000000 00:00 0
MAP: 7f2665200000-7f2665400000 rw-p 00000000 00:00 0
MAP: 7f266540f000-7f266541f000 rwxp 00000000 00:00 0
MAP: 7f266542f000-7f266543f000 rwxp 00000000 00:00 0
MAP: 7f266546f000-7f266547f000 rwxp 00000000 00:00 0
MAP: 7f26654ff000-7f266551f000 rwxp 00000000 00:00 0
MAP: 7f266552f000-7f266554f000 rwxp 00000000 00:00 0
MAP: 7f26655ef000-7f26655ff000 rwxp 00000000 00:00 0
MAP: 7f26655ff000-7f2665600000 ---p 00000000 00:00 0
MAP: 7f2665600000-7f2665800000 rw-p 00000000 00:00 0
MAP: 7f2665800000-7f2665a00000 rw-p 00000000 00:00 0
MAP: 7f2665a0f000-7f2665a1f000 rwxp 00000000 00:00 0
MAP: 7f2665a3f000-7f2665a5f000 rwxp 00000000 00:00 0
MAP: 7f2665a6f000-7f2665a7f000 rwxp 00000000 00:00 0
MAP: 7f2665abf000-7f2665acf000 rwxp 00000000 00:00 0
MAP: 7f2665aef000-7f2665b0f000 rwxp 00000000 00:00 0
MAP: 7f2665b2f000-7f2665b5f000 rwxp 00000000 00:00 0
MAP: 7f2665b8f000-7f2665b9f000 rwxp 00000000 00:00 0
MAP: 7f2665bff000-7f2665c00000 ---p 00000000 00:00 0
MAP: 7f2665c00000-7f2665e00000 rw-p 00000000 00:00 0
MAP: 7f2665e00000-7f2666000000 rw-p 00000000 00:00 0
MAP: 7f266611f000-7f26661ff000 rwxp 00000000 00:00 0
MAP: 7f26661ff000-7f2666200000 ---p 00000000 00:00 0
MAP: 7f2666200000-7f2666400000 rw-p 00000000 00:00 0
MAP: 7f2666400000-7f2666a00000 rw-p 00000000 00:00 0
MAP: 7f2666a0d000-7f2666bfd000 rwxp 00000000 00:00 0
MAP: 7f2666bfd000-7f2666bfe000 ---p 00000000 00:00 0
MAP: 7f2666bfe000-7f2666dfe000 rw-p 00000000 00:00 0
MAP: 7f2666dfe000-7f2666dff000 ---p 00000000 00:00 0
MAP: 7f2666dff000-7f2666fff000 rw-p 00000000 00:00 0
MAP: 7f2666fff000-7f2667000000 ---p 00000000 00:00 0
MAP: 7f2667000000-7f2667200000 rw-p 00000000 00:00 0
MAP: 7f2667200000-7f2667400000 rw-p 00000000 00:00 0
MAP: 7f266740f000-7f26675ff000 rwxp 00000000 00:00 0
MAP: 7f26675ff000-7f2667600000 ---p 00000000 00:00 0
MAP: 7f2667600000-7f2667800000 rw-p 00000000 00:00 0
MAP: 7f2667800000-7f2667c00000 rw-p 00000000 00:00 0
MAP: 7f2667c0e000-7f2667dfe000 rwxp 00000000 00:00 0
MAP: 7f2667dfe000-7f2667dff000 ---p 00000000 00:00 0
MAP: 7f2667dff000-7f2667fff000 rw-p 00000000 00:00 0
MAP: 7f2667fff000-7f2668000000 ---p 00000000 00:00 0
MAP: 7f2668000000-7f2668200000 rw-p 00000000 00:00 0
MAP: 7f2668200000-7f266ac00000 rw-p 00000000 00:00 0
MAP: 7f266ac0f000-7f266adff000 rwxp 00000000 00:00 0
MAP: 7f266adff000-7f266ae00000 ---p 00000000 00:00 0
MAP: 7f266ae00000-7f266b000000 rw-p 00000000 00:00 0
MAP: 7f266b000000-7f266b400000 rw-p 00000000 00:00 0
MAP: 7f266b40f000-7f266b5ff000 rwxp 00000000 00:00 0
MAP: 7f266b5ff000-7f266b600000 ---p 00000000 00:00 0
MAP: 7f266b600000-7f266b800000 rw-p 00000000 00:00 0
MAP: 7f266b800000-7f266bc00000 rw-p 00000000 00:00 0
MAP: 7f266bc05000-7f266bc95000 rwxp 00000000 00:00 0
MAP: 7f266bc95000-7f266bc96000 ---p 00000000 00:00 0
MAP: 7f266bc96000-7f266bc9f000 rw-p 00000000 00:00 0
MAP: 7f266bc9f000-7f266bca0000 ---p 00000000 00:00 0
MAP: 7f266bca0000-7f266be00000 rwxp 00000000 00:00 0
MAP: 7f266be00000-7f266c200000 rw-p 00000000 00:00 0
MAP: 7f266c20a000-7f266c33a000 rwxp 00000000 00:00 0
MAP: 7f266c33a000-7f266c33f000 rw-p 00000000 00:00 0
MAP: 7f266c33f000-7f266c502000 r-xp 00000000 103:01 2457 /lib64/libc-2.17.so
MAP: 7f266c502000-7f266c701000 ---p 001c3000 103:01 2457 /lib64/libc-2.17.so
MAP: 7f266c701000-7f266c705000 r--p 001c2000 103:01 2457 /lib64/libc-2.17.so
MAP: 7f266c705000-7f266c707000 rw-p 001c6000 103:01 2457 /lib64/libc-2.17.so
MAP: 7f266c707000-7f266c70c000 rw-p 00000000 00:00 0
MAP: 7f266c70c000-7f266c723000 r-xp 00000000 103:01 2483 /lib64/libpthread-2.17.so
MAP: 7f266c723000-7f266c922000 ---p 00017000 103:01 2483 /lib64/libpthread-2.17.so
MAP: 7f266c922000-7f266c923000 r--p 00016000 103:01 2483 /lib64/libpthread-2.17.so
MAP: 7f266c923000-7f266c924000 rw-p 00017000 103:01 2483 /lib64/libpthread-2.17.so
MAP: 7f266c924000-7f266c928000 rw-p 00000000 00:00 0
MAP: 7f266c928000-7f266ca29000 r-xp 00000000 103:01 2465 /lib64/libm-2.17.so
MAP: 7f266ca29000-7f266cc28000 ---p 00101000 103:01 2465 /lib64/libm-2.17.so
MAP: 7f266cc28000-7f266cc29000 r--p 00100000 103:01 2465 /lib64/libm-2.17.so
MAP: 7f266cc29000-7f266cc2a000 rw-p 00101000 103:01 2465 /lib64/libm-2.17.so
MAP: 7f266cc2a000-7f266cc55000 r-xp 00000000 103:01 525505 /opt/splunk/lib/libbson-1.0.so.0.0.0
MAP: 7f266cc55000-7f266cc58000 rw-p 0002b000 103:01 525505 /opt/splunk/lib/libbson-1.0.so.0.0.0
MAP: 7f266cc58000-7f266cc5e000 rw-p 00000000 00:00 0
MAP: 7f266cc5e000-7f266ccba000 r-xp 00000000 103:01 525518 /opt/splunk/lib/libmongoc-1.0.so.0.0.0
MAP: 7f266ccba000-7f266ccbc000 rw-p 0005b000 103:01 525518 /opt/splunk/lib/libmongoc-1.0.so.0.0.0
MAP: 7f266ccbc000-7f266cdb3000 r-xp 00000000 103:01 525523 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f266cdb3000-7f266cdb7000 rw-p 000f6000 103:01 525523 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 7f266cdb7000-7f266cdb8000 rw-p 00000000 00:00 0
MAP: 7f266cdb8000-7f266ce59000 r-xp 00000000 103:01 525502 /opt/splunk/lib/libarchive.so.13.3.1
MAP: 7f266ce59000-7f266ce5d000 rw-p 000a1000 103:01 525502 /opt/splunk/lib/libarchive.so.13.3.1
MAP: 7f266ce5d000-7f266ce5f000 r-xp 00000000 103:01 2463 /lib64/libdl-2.17.so
MAP: 7f266ce5f000-7f266d05f000 ---p 00002000 103:01 2463 /lib64/libdl-2.17.so
MAP: 7f266d05f000-7f266d060000 r--p 00002000 103:01 2463 /lib64/libdl-2.17.so
MAP: 7f266d060000-7f266d061000 rw-p 00003000 103:01 2463 /lib64/libdl-2.17.so
MAP: 7f266d061000-7f266d30d000 r-xp 00000000 103:01 525510 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f266d30d000-7f266d339000 rw-p 002ac000 103:01 525510 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 7f266d339000-7f266d33d000 rw-p 00000000 00:00 0
MAP: 7f266d33d000-7f266d393000 r-xp 00000000 103:01 525531 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 7f266d393000-7f266d397000 rw-p 00055000 103:01 525531 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 7f266d397000-7f266d398000 rw-p 00000000 00:00 0
MAP: 7f266d398000-7f266d413000 r-xp 00000000 103:01 525534 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 7f266d413000-7f266d417000 rw-p 0007a000 103:01 525534 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 7f266d417000-7f266d5d0000 r-xp 00000000 103:01 525528 /opt/splunk/lib/libxml2.so.2.9.8
MAP: 7f266d5d0000-7f266d5da000 rw-p 001b8000 103:01 525528 /opt/splunk/lib/libxml2.so.2.9.8
MAP: 7f266d5da000-7f266d5db000 rw-p 00000000 00:00 0
MAP: 7f266d5db000-7f266d5e2000 r-xp 00000000 103:01 2487 /lib64/librt-2.17.so
MAP: 7f266d5e2000-7f266d7e1000 ---p 00007000 103:01 2487 /lib64/librt-2.17.so
MAP: 7f266d7e1000-7f266d7e2000 r--p 00006000 103:01 2487 /lib64/librt-2.17.so
MAP: 7f266d7e2000-7f266d7e3000 rw-p 00007000 103:01 2487 /lib64/librt-2.17.so
MAP: 7f266d7e3000-7f266d805000 r-xp 00000000 103:01 2450 /lib64/ld-2.17.so
MAP: 7f266d805000-7f266d806000 rw-p 00000000 00:00 0
MAP: 7f266d806000-7f266d821000 r-xp 00000000 103:01 525540 /opt/splunk/lib/libz.so.1.2.11
MAP: 7f266d821000-7f266d822000 rw-p 0001b000 103:01 525540 /opt/splunk/lib/libz.so.1.2.11
MAP: 7f266d822000-7f266d823000 rw-p 00000000 00:00 0
MAP: 7f266d823000-7f266d834000 r-xp 00000000 103:01 525508 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f266d834000-7f266d836000 rw-p 00010000 103:01 525508 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 7f266d836000-7f266d838000 rw-p 00000000 00:00 0
MAP: 7f266d838000-7f266d8a4000 r-xp 00000000 103:01 525525 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f266d8a4000-7f266d8ae000 rw-p 0006b000 103:01 525525 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7f266d8ae000-7f266d900000 r-xp 00000000 103:01 525537 /opt/splunk/lib/libxslt.so.1.1.30
MAP: 7f266d900000-7f266d902000 rw-p 00052000 103:01 525537 /opt/splunk/lib/libxslt.so.1.1.30
MAP: 7f266d902000-7f266d903000 rw-p 00000000 00:00 0
MAP: 7f266d903000-7f266d9aa000 r-xp 00000000 103:01 525519 /opt/splunk/lib/libpcre2-8.so
MAP: 7f266d9aa000-7f266d9ab000 rw-p 000a7000 103:01 525519 /opt/splunk/lib/libpcre2-8.so
MAP: 7f266d9ab000-7f266d9ac000 rw-p 00000000 00:00 0
MAP: 7f266d9b0000-7f266d9b1000 rw-p 00000000 00:00 0
MAP: 7f266d9b1000-7f266d9ff000 r-xp 00000000 103:01 525515 /opt/splunk/lib/libjemalloc.so.2
MAP: 7f266d9ff000-7f266da02000 rw-p 0004d000 103:01 525515 /opt/splunk/lib/libjemalloc.so.2
MAP: 7f266da02000-7f266da04000 rw-p 00000000 00:00 0
MAP: 7f266da04000-7f266da05000 r--p 00021000 103:01 2450 /lib64/ld-2.17.so
MAP: 7f266da05000-7f266da06000 rw-p 00022000 103:01 2450 /lib64/ld-2.17.so
MAP: 7f266da06000-7f266da07000 rw-p 00000000 00:00 0
MAP: 7ffeb6b34000-7ffeb6b55000 rw-p 00000000 00:00 0 [stack]
MAP: 7ffeb6b5c000-7ffeb6b5f000 r--p 00000000 00:00 0 [vvar]
MAP: 7ffeb6b5f000-7ffeb6b61000 r-xp 00000000 00:00 0 [vdso]
MAP: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Last errno: 0
Threads running: 74
Runtime: 167584.190897s
argv: [splunkd -p 8089 start]
Regex JIT enabled

using CLOCK_MONOTONIC
Thread: "TcpChannelThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f2648130890:
00000000 00 f7 1f c5 25 7f 00 00 |....%...|
00000008
commandForThread=0, nextIdle=0x7f26482c1380, requestAfterThread=0, tpfd=0x7f2636ce2800, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f26481309f0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil)
writeDataAvail @0x7f2648130a50: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=(nil)
wbuf: ptr=0x7f2648130af0, size=0x8000, rptr=0x0, wptr=0x0
HttpListeningConnection: _transactionActive=Y, _haveHadTransaction=Y, _alreadyLoggedTimeout=N
HttpTcpConnection: peer=172.31.40.243, _desiredCompressionLevel=6
RestHttpServerTransaction: _restPath="search/jobs/katekate_QUxURVNTX0RFTU8search1_1533147731.70004/results_preview", namespaced=N, context=-/-, session=[user=kate, refcnt=4, touched=1533147999, refreshEligible=1533148038, removed=N, id=4039104a0af8615b9848dbec1cb3811f, created=1533147588, refreshed=1533147813, expires=1533151413, initialLife=3600, createdBy=41544C65-F692-4316-A695-36D0A372FD96, ip=172.31.40.243, csrf=1039714564027896298]
HttpServerTransaction: _state=6, _shouldLog=Y, _startTime=1533147999.545457505455
REQUEST: GET /en-US/splunkd/raw/services/search/jobs/katekate_QUxURVNTX0RFTU8search1_1533147731.70004/results_preview?output_mode=json_cols&count=100&offset=0&search=geofilter%20south%3D-57.32652122521708%20west%3D-180%20north%3D57.326521225217064%20east%3D180%20maxclusters%3D100&show_metadata=true&=1533147729757 HTTP/1.1
Host: splunkdemo.omnicrom.internal:8000
Connection: keep-alive
Accept: text/javascript, text/html, application/xml, text/xml, /
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
X-Requested-With: XMLHttpRequest
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: splunkweb_csrf_token_8000=1039714564027896298; session_id_8000=76e3a35e69ca0ec250c919ecfccce1ef674d328c; token_key=1039714564027896298; experience_id=4fdc9c00-9181-a707-bb00-cad149935d77; splunkd_8000=W5FcjsPDUHz2XafpNk^wJFTntmYDl9tHZ1gxSCwRArgUs0mvG67^FtEiAGAD2WLKyULea8RRapvo6^HNg7Y1UgnZk1cjEfWEsEVQDow67PxPzCCBofT3Yb8vwiLuf8CL
_bytesReceived=0, _maximumRequestDataSize=0, _totalBytesExpectedOfRequestData=-1
_bytesLeftInRequestDataChunk=0, _requestTransferEncodingIsChunked=N, _receivingRequestDataForever=N
_needToSetupRequestGunzip=N, _owedConsume=139802395200624, _wantSavedRequestData=N
_100continue=0, _expectDisconnect=N, _overrideSourceState=0
POST arguments: {}
REPLY: 200
Set-Cookie: splunkd_8000=W5FcjsPDUHz2XafpNk^wJFTntmYDl9tHZ1gxSCwRArgUs0mvG67^FtEiAGAD2WLKyULea8RRapvo6^HNg7Y1UgnZk1cjEfWEsEVQDow67PxPzCCBofT3Yb8vwiLuf8CL; Path=/; Secure; HttpOnly; Max-Age=3600; Expires=Wed, 01 Aug 2018 19:26:39 GMT
Set-Cookie: splunkweb_csrf_token_8000=1039714564027896298; Path=/; Secure; Max-Age=157680000; Expires=Mon, 31 Jul 2023 18:26:39 GMT
ServicesEndpointReplyDataProvider: _setupState=0, _outputMode=32, _explicitOutputMode=Y
GET args: {["count"] = "100", ["offset"] = "0", ["search"] = "geofilter south=-57.32652122521708 west=-180 north=57.326521225217064 east=180 maxclusters=100", ["show_metadata"] = "true"}
_allowedMethods={GET,POST,PUT,DELETE,HEAD,OPTIONS}, _preconditionState=0
_wantsSeparateThread=N, _alreadyBuiltHeaders=N, _needToSendBody=Y
_bodyBytesWritten=0, _chunkedState=0, _isLastTransaction=N
_varyBy=0x8, _redirectUrl="", _downloadFilename="", _totalScheduledLength=0
_willSendDataLater=N, _toSendState=0, _toSendSafe=Y
_knowCompleteLength=N, _desiredCompressionLevel=6
_replyIsGzipCompressed=N, _cacheControl=0x0, _maxCacheSeconds=4294967295, _dontIncludeFrameOptions=N
In TcpChannel 0x7f2636ce2800, _tcloop=0x7f2646f7b288, no async write data, _data._shouldKill=N, r/w_timeouts=5.000/300.000, timeout_count=0
SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="ECDHE-RSA-AES128-GCM-SHA256", compression="none"
rbuf: ptr=0x7f2636ce28a0, size=0x2000, rptr=0x0, wptr=0x0
TcpChannelAcceptor: , tcloop=0x7f2646f7b288, _disabledReasons=0, _activeCount=9, _inflightSubordinateAccepts=0
HttpListener: ssl=Y, _maxActiveConnections=21845, _wellBelowConnectionLimit=Y, _maxThreads=5191
SplunkdHttpListener: PORT: _allowGzip=Y, bind=https://:8000
conf: _sslopt={rootCAPath="", caCertFile="", certFile="/opt/splunk/etc/auth/splunkweb/cert.pem", privateKeyFile="/opt/splunk/etc/auth/splunkweb/privkey.pem", privateKeyPassword_set=N, commonNameToCheck="", altNameToCheck="", allowSslRenegotiation=Y, sslVersions="TLS1.2", cipherSuite="ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256", ecdhCurves="prime256v1, secp384r1, secp521r1", useCompression=N, quietShutdown=NdhFile="", shouldVerifyClientCert=N}, _allowSslRenegotiation=Y, _frameOptionsSameOrigin=Y, _strictTransportSecurityHeader=N, _allowBasicAuth=N, _allowCookieAuth=N
conf: _streamInWriteTimeout=5.000, _maxContentLength=524288000, _maxThreads=5191, _maxSockets=21845, _forceHttp10=0
_thread=0x7f2648130880: commandForThread=0, nextIdle=0x7f26482c1380, requestAfterThread=0, _tpfd=0x7f2636ce2800, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f26481309f0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil)
writeDataAvail @0x7f2648130a50: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=(nil)
wbuf: ptr=0x7f2648130af0, size=0x8000, rptr=0x0, wptr=0x0

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 00050653 02100800 FFFA3203 1F8BFBFF
2: 76036301 00F0B5FF 00000000 00C30000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000006 00000000 00000001 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 00000000 00000000 00000000 00000000
B: 00000000 00000000 00000000 00000000
C: 00000000 00000000 00000000 00000000
😧 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000121 2C100800
80000002: 00000000 00000000 00000000 00000000
80000003: 00000000 00000000 00000000 00000000
80000004: 00000000 00000000 00000000 00000000
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 0000302E 00000000 00000000 00000000
terminating...

[Janssen135]

in Splunk Version 8.1.6 the same issue is still there.

Received fatal signal 11 (Segmentation fault). Cause: No memory mapped at address [0x00007F5F02649C08]. Crashing thread: BatchSearch

Received fatal signal 11 (Segmentation fault). Cause: No memory mapped at address [0x00007F7262188008]. Crashing thread: phase_1

What is the current solution?

[dvg06]

Support says this is known issue (SPL-153976) and fixed as part of 7.1.3 (https://docs.splunk.com/Documentation/Splunk/7.1.3/ReleaseNotes/Fixedissues), I am upgrading tonight to check if it works.

[mbedin]

Same thing on 7.1.2 on CentOS Linux release 7.5.1804 (Core), all 3 nodes of a Search Head Cluster crashed almost simultaneously!

[scc00]

Yeah I am still dealing with this issue. Did you figure out your issue?

[fboeje]

Same here, one or more searchheads crash several times a day, After recent upgrade to splunk 7.2. Support Call is already in progress. No solution so far. As a workaround we just restart them.

[gjanders]

7.1.4 is out, it might be worth a try...

[mbedin]

Support recommended me to update to 7.1.3.

[jethompson_splu]

@scc00 -- Please submit a Support Case for this issue as we have recently had several reports very similar to this one. This is something that our Developers will need to look into.

My first thought on this issue as its a SIG-11 SegFault is that the System ran out of available Memory (RAM) and the Kernel Killed off this Search Process to prevent a Kernel Panic. You may be able to overcome this issue if it is Kernel Related by updating the Linux Kernel in use on that AWS Linux AMI. If possible as a Linux Administrator what I would do is setup a CentOS instance to see if this issue is present with that Linux Distro.

From the "Crash Report" provided I can see that the Splunk process that was "killed" was a Search. This was seen from the following line of the Printout you provided:

[0x0000556D51DB0820] ZN16SearchResultsMem23appendSharedDestructiveERS + 208 (splunkd + 0x1029820)

This is showing that a Search appears to have attempted to update its Memory Location with some new data and could not complete that process. We will need to have you submit a Support Case to have this issue reviewed by the Splunk Developers.

Thank you,

[scc00]

So I found an error within /var/log/messages that mirrored the time of the crash. It seems NTP was out of sync and kept terminating on the OS. So i removed NTP and switched over to AWS chrony and so far so good. I will update this thread in a few days when I am certain this was the resolution.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

[jethompson_splu]

@scc00 -- Thank you for that information and that sounds like a Really BAD Bug on the AWS Linux AMI side of things if NTP is not viable for use. I would reach out to AWS as NTP is a Default Linux Tool for Syncing Time and should be able to be used in the AWS Linux AMI without causing a SIG-11 SegFault in an application that requires Time to be Synced between instances.

The fact that you had to remove NTP and change to using Chrony would suggest that AWS may have introduced a Bug into their Kernel that prevents NTP from running and forcing their users to use the "In-House" AWS Time Sync Services. This should be "Good-2-Go" for use but if there is ever a Time Sync issue at AWS then this type of issue could return.

It might be worth testing another Linux Distro for your Environment such as CentOS, Ubuntu, RHEL (RedHat), or SuSE Linux. I do know that these Linux Distros work with NTP "Out-of-the-Box" at AWS.

Again thank you for that info, I will add it to my "Potential Causes" Bag of tricks.

[scc00]

So the new test box crashed twice after fixing the NTP error. Now it's crashing with the below error in /var/log/messages/

Aug 16 18:07:24 ip-XXX-XX-XX-XXX kernel: [607367.714852] splunkd[16873]: segfault at 10 ip 000055be8e253820 sp 00007fe53fdfcb30 error 6 in splunkd[55be8d22a000+27bc000]
Aug 17 07:53:30 ip-XXX-XX-XX-XXX kernel: [656933.465603] splunkd[30552]: segfault at 10 ip 0000563f6170f820 sp 00007f2106ffbb30 error 6 in splunkd[563f606e6000+27bc000]

and this in splunkd.log:
[build a0c72a66db66] 2018-08-17 07:53:30
Received fatal signal 11 (Segmentation fault).
Cause:
No memory mapped at address [0x0000000000000010].
Crashing thread: TcpChannelThread

[scc00]

So Splunk told me to adjust the CPU and clean up some skipped searches and limits.conf. Implemented all of that but now the instance still crashes but now i'm seeing the CPU spiking caused by search and scripted input. Then the instance crashes again with the same error above.