Splunk crashes when trying to install an app from "Browse more apps" section

pyde88 — Tue, 29 Sep 2020 16:11:30 GMT

Our splunk is running on RHEL 7 as a non-root user. Splunk is behind firewall and i configured proxy settings. As soon as i enter my splunk credentials for installing any app in "Browse more apps" section, splunk gets crashed everytime. I see following in crash log.

Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 18714 running under UID 40586.
Crashing thread: TcpChannelThread
Registers:
RIP: [0x00007F436326C1F7] gsignal + 55 (libc.so.6 + 0x351F7)
RDI: [0x000000000000491A]
RSI: [0x0000000000005DA9]
RBP: [0x00007F43633B7E68]
RSP: [0x00007F43475FCFC8]
RAX: [0x0000000000000000]
RBX: [0x00007F4364879000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000148]
R9: [0xFEFEFEFF092D6364]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x0000559FD2F96158]
R13: [0x0000559FD3074D40]
R14: [0x00007F43475FD1F0]
R15: [0x00007F43620F786A]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007F436326C1F7] gsignal + 55 (libc.so.6 + 0x351F7)
[0x00007F436326D8E8] abort + 328 (libc.so.6 + 0x368E8)
[0x00007F4363265266] ? (libc.so.6 + 0x2E266)
[0x00007F4363265312] ? (libc.so.6 + 0x2E312)
[0x0000559FD22C31C4] ZN21HttpClientTransaction15_handleRedirectEv + 852 (splunkd + 0x11B31C4)
[0x0000559FD22C3932] _ZN21HttpClientTransaction18_finishTransactionEv + 354 (splunkd + 0x11B3932)
[0x0000559FD22C47C3] _ZN20HttpClientConnection10parseReplyEPKcS1 + 1747 (splunkd + 0x11B47C3)
[0x0000559FD22C58C0] _ZN20HttpClientConnection13dataAvailableEv + 160 (splunkd + 0x11B58C0)
[0x0000559FD2360B88] _ZN11TcpOutbound6_do_ioE18PollableDescriptor + 440 (splunkd + 0x1250B88)
[0x0000559FD2361C01] _ZN11TcpOutbound11when_eventsE18PollableDescriptor + 33 (splunkd + 0x1251C01)
[0x0000559FD22AA9B6] _ZN8PolledFd8do_eventEv + 134 (splunkd + 0x119A9B6)
[0x0000559FD22AB8BB] _ZN9EventLoop3runEv + 651 (splunkd + 0x119B8BB)
[0x0000559FD23623B0] _ZN15TcpOutboundLoop3runEv + 16 (splunkd + 0x12523B0)
[0x0000559FD22C1712] _ZN21HttpClientTransaction22runSyncAndShutdownLoopEP15TcpOutboundLoop + 50 (splunkd + 0x11B1712)
[0x0000559FD22C17D5] _ZN21HttpClientTransaction7runSyncEv + 69 (splunkd + 0x11B17D5)
[0x0000559FD2235B41] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR28ApplicationUpdateTransaction + 145 (splunkd + 0x1125B41)
[0x0000559FD2235FB5] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR8Pathnameb + 357 (splunkd + 0x1125FB5)
[0x0000559FD20FF3AC] _ZN21LocalAppsAdminHandler13handleInstallER10ConfigInfoPK3StrS4_b + 2652 (splunkd + 0xFEF3AC)
[0x0000559FD20FF9ED] _ZN21LocalAppsAdminHandler12handleCreateER10ConfigInfo + 253 (splunkd + 0xFEF9ED)
[0x0000559FD1DF980C] _ZN14MConfigHandler14executeHandlerER10ConfigInfo + 620 (splunkd + 0xCE980C)
[0x0000559FD1E09C7D] _ZN14MConfigHandler2goER10ConfigInfo + 189 (splunkd + 0xCF9C7D)
[0x0000559FD1E0A844] _ZN29AdminManagerReplyDataProvider2goEv + 804 (splunkd + 0xCFA844)
[0x0000559FD1EA3078] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0xD93078)
[0x0000559FD1E98B2F] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 31 (splunkd + 0xD88B2F)
[0x0000559FD22D9FE0] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 272 (splunkd + 0x11C9FE0)
[0x0000559FD19180B3] _ZN16TcpChannelThread4mainEv + 227 (splunkd + 0x8080B3)
[0x0000559FD2363440] _ZN6Thread8callMainEPv + 64 (splunkd + 0x1253440)
[0x00007F4363601E25] ? (libpthread.so.0 + 0x7E25)
[0x00007F436332F34D] clone + 109 (libc.so.6 + 0xF834D)
Linux / gssit-devops-qa / 3.10.0-693.1.1.el7.x86_64 / #1 SMP Thu Aug 3 08:15:31 EDT 2017 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2017-10-05 16:38:25.291 -0400 Interrupt signal received
2017-10-05 16:40:18.548 -0400 splunkd started (build 4b804538c686)
2017-10-05 16:44:46.866 -0400 Interrupt signal received
2017-10-05 16:46:48.163 -0400 splunkd started (build 4b804538c686)
2017-10-05 22:55:50.030 -0400 Interrupt signal received
2017-10-05 22:57:42.807 -0400 splunkd started (build 4b804538c686)
splunkd: /home/build/build-src/kimono/src/util/HttpClientRequest.cpp:1860: void HttpClientTransaction::_handleRedirect(): Assertion _redirectReply == REPLY_EATING_NORMAL' failed. 2017-10-05 23:02:25.471 -0400 splunkd started (build 4b804538c686) splunkd: /home/build/build-src/kimono/src/util/HttpClientRequest.cpp:1860: void HttpClientTransaction::_handleRedirect(): Assertion_redirectReply == REPLY_EATING_NORMAL' failed.

Re: Splunk crashes when trying to install an app from "Browse more apps" section

lfedak_splunk — Sat, 07 Oct 2017 00:51:17 GMT

Hey @pyde88 - Can you let us know which apps specifically?

Re: Splunk crashes when trying to install an app from "Browse more apps" section

pyde88 — Mon, 09 Oct 2017 13:52:35 GMT

I am getting this for any app. In particualr, I tried Github Addon and Monitoring Kubernetes etc.

Re: Splunk crashes when trying to install an app from "Browse more apps" section

templier — Mon, 09 Oct 2017 14:24:27 GMT

So, how about install using .tar.gz file downloaded from splunkbase ?
You go Splunk->Manage apps->Install app from file and specify the path to the archive.

And check have you splunk user access to write data to splunk_home directory?

Re: Splunk crashes when trying to install an app from "Browse more apps" section

ssmiesko — Tue, 19 Jun 2018 16:42:32 GMT

I'm receiving this same crash dump on 7.1.0 in a docker container.

I've set up my Proxy and RootCA settings to access the internet and download apps/content. Yes, we can certainly install from tar.gz packages downloaded from splunkbase, but that's not an appropriate answer to a hard crash like this.

Hopefully someone can investigate further.

Re: Splunk crashes when trying to install an app from "Browse more apps" section

lacastillo — Thu, 28 Jun 2018 19:10:18 GMT

You could also download the .tar /.gz file directly from splunkbase.com and extract it to $SPLUNK_HOME/etc/apps
then restart Splunk.

Re: Splunk crashes when trying to install an app from "Browse more apps" section

ictsoc — Tue, 09 Apr 2019 08:17:24 GMT

Hello,

I realise this is an older question, but our Splunk search head just crashed with the exact same crash log when trying to install an app from 'browse more apps'. Also the setup is very similar: RHEL7, non root user, corporate proxy configured.

Did you ever manage to get this resolved?

Kr,

Jerome

topic Re: Splunk crashes when trying to install an app from "Browse more apps" section in Deployment Architecture

Splunk crashes when trying to install an app from "Browse more apps" section

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Re: Splunk crashes when trying to install an app from "Browse more apps" section