Splunk crashes when trying to install an app from "Browse more apps" section

[pyde88]

Our splunk is running on RHEL 7 as a non-root user. Splunk is behind firewall and i configured proxy settings. As soon as i enter my splunk credentials for installing any app in "Browse more apps" section, splunk gets crashed everytime. I see following in crash log.

Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 18714 running under UID 40586.
Crashing thread: TcpChannelThread
Registers:
RIP: [0x00007F436326C1F7] gsignal + 55 (libc.so.6 + 0x351F7)
RDI: [0x000000000000491A]
RSI: [0x0000000000005DA9]
RBP: [0x00007F43633B7E68]
RSP: [0x00007F43475FCFC8]
RAX: [0x0000000000000000]
RBX: [0x00007F4364879000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000148]
R9: [0xFEFEFEFF092D6364]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x0000559FD2F96158]
R13: [0x0000559FD3074D40]
R14: [0x00007F43475FD1F0]
R15: [0x00007F43620F786A]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007F436326C1F7] gsignal + 55 (libc.so.6 + 0x351F7)
[0x00007F436326D8E8] abort + 328 (libc.so.6 + 0x368E8)
[0x00007F4363265266] ? (libc.so.6 + 0x2E266)
[0x00007F4363265312] ? (libc.so.6 + 0x2E312)
[0x0000559FD22C31C4] ZN21HttpClientTransaction15_handleRedirectEv + 852 (splunkd + 0x11B31C4)
[0x0000559FD22C3932] _ZN21HttpClientTransaction18_finishTransactionEv + 354 (splunkd + 0x11B3932)
[0x0000559FD22C47C3] _ZN20HttpClientConnection10parseReplyEPKcS1 + 1747 (splunkd + 0x11B47C3)
[0x0000559FD22C58C0] _ZN20HttpClientConnection13dataAvailableEv + 160 (splunkd + 0x11B58C0)
[0x0000559FD2360B88] _ZN11TcpOutbound6_do_ioE18PollableDescriptor + 440 (splunkd + 0x1250B88)
[0x0000559FD2361C01] _ZN11TcpOutbound11when_eventsE18PollableDescriptor + 33 (splunkd + 0x1251C01)
[0x0000559FD22AA9B6] _ZN8PolledFd8do_eventEv + 134 (splunkd + 0x119A9B6)
[0x0000559FD22AB8BB] _ZN9EventLoop3runEv + 651 (splunkd + 0x119B8BB)
[0x0000559FD23623B0] _ZN15TcpOutboundLoop3runEv + 16 (splunkd + 0x12523B0)
[0x0000559FD22C1712] _ZN21HttpClientTransaction22runSyncAndShutdownLoopEP15TcpOutboundLoop + 50 (splunkd + 0x11B1712)
[0x0000559FD22C17D5] _ZN21HttpClientTransaction7runSyncEv + 69 (splunkd + 0x11B17D5)
[0x0000559FD2235B41] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR28ApplicationUpdateTransaction + 145 (splunkd + 0x1125B41)
[0x0000559FD2235FB5] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR8Pathnameb + 357 (splunkd + 0x1125FB5)
[0x0000559FD20FF3AC] _ZN21LocalAppsAdminHandler13handleInstallER10ConfigInfoPK3StrS4_b + 2652 (splunkd + 0xFEF3AC)
[0x0000559FD20FF9ED] _ZN21LocalAppsAdminHandler12handleCreateER10ConfigInfo + 253 (splunkd + 0xFEF9ED)
[0x0000559FD1DF980C] _ZN14MConfigHandler14executeHandlerER10ConfigInfo + 620 (splunkd + 0xCE980C)
[0x0000559FD1E09C7D] _ZN14MConfigHandler2goER10ConfigInfo + 189 (splunkd + 0xCF9C7D)
[0x0000559FD1E0A844] _ZN29AdminManagerReplyDataProvider2goEv + 804 (splunkd + 0xCFA844)
[0x0000559FD1EA3078] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0xD93078)
[0x0000559FD1E98B2F] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 31 (splunkd + 0xD88B2F)
[0x0000559FD22D9FE0] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 272 (splunkd + 0x11C9FE0)
[0x0000559FD19180B3] _ZN16TcpChannelThread4mainEv + 227 (splunkd + 0x8080B3)
[0x0000559FD2363440] _ZN6Thread8callMainEPv + 64 (splunkd + 0x1253440)
[0x00007F4363601E25] ? (libpthread.so.0 + 0x7E25)
[0x00007F436332F34D] clone + 109 (libc.so.6 + 0xF834D)
Linux / gssit-devops-qa / 3.10.0-693.1.1.el7.x86_64 / #1 SMP Thu Aug 3 08:15:31 EDT 2017 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2017-10-05 16:38:25.291 -0400 Interrupt signal received
2017-10-05 16:40:18.548 -0400 splunkd started (build 4b804538c686)
2017-10-05 16:44:46.866 -0400 Interrupt signal received
2017-10-05 16:46:48.163 -0400 splunkd started (build 4b804538c686)
2017-10-05 22:55:50.030 -0400 Interrupt signal received
2017-10-05 22:57:42.807 -0400 splunkd started (build 4b804538c686)
splunkd: /home/build/build-src/kimono/src/util/HttpClientRequest.cpp:1860: void HttpClientTransaction::_handleRedirect(): Assertion _redirectReply == REPLY_EATING_NORMAL' failed.
2017-10-05 23:02:25.471 -0400 splunkd started (build 4b804538c686)
splunkd: /home/build/build-src/kimono/src/util/HttpClientRequest.cpp:1860: void HttpClientTransaction::_handleRedirect(): Assertion_redirectReply == REPLY_EATING_NORMAL' failed.

[ictsoc]

Hello,

I realise this is an older question, but our Splunk search head just crashed with the exact same crash log when trying to install an app from 'browse more apps'. Also the setup is very similar: RHEL7, non root user, corporate proxy configured.

Did you ever manage to get this resolved?

Kr,

Jerome

[lacastillo]

You could also download the .tar /.gz file directly from splunkbase.com and extract it to $SPLUNK_HOME/etc/apps
then restart Splunk.

[ssmiesko]

I'm receiving this same crash dump on 7.1.0 in a docker container.

I've set up my Proxy and RootCA settings to access the internet and download apps/content. Yes, we can certainly install from tar.gz packages downloaded from splunkbase, but that's not an appropriate answer to a hard crash like this.

Hopefully someone can investigate further.

[templier]

So, how about install using .tar.gz file downloaded from splunkbase ?
You go Splunk->Manage apps->Install app from file and specify the path to the archive.

And check have you splunk user access to write data to splunk_home directory?

[lfedak_splunk]

Hey @pyde88 - Can you let us know which apps specifically?

[pyde88]

I am getting this for any app. In particualr, I tried Github Addon and Monitoring Kubernetes etc.