Deployment Architecture
Highlighted

Splunk crashes when trying to install an app from "Browse more apps" section

Engager

Our splunk is running on RHEL 7 as a non-root user. Splunk is behind firewall and i configured proxy settings. As soon as i enter my splunk credentials for installing any app in "Browse more apps" section, splunk gets crashed everytime. I see following in crash log.

Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 18714 running under UID 40586.
Crashing thread: TcpChannelThread
Registers:
RIP: [0x00007F436326C1F7] gsignal + 55 (libc.so.6 + 0x351F7)
RDI: [0x000000000000491A]
RSI: [0x0000000000005DA9]
RBP: [0x00007F43633B7E68]
RSP: [0x00007F43475FCFC8]
RAX: [0x0000000000000000]
RBX: [0x00007F4364879000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000148]
R9: [0xFEFEFEFF092D6364]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x0000559FD2F96158]
R13: [0x0000559FD3074D40]
R14: [0x00007F43475FD1F0]
R15: [0x00007F43620F786A]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007F436326C1F7] gsignal + 55 (libc.so.6 + 0x351F7)
[0x00007F436326D8E8] abort + 328 (libc.so.6 + 0x368E8)
[0x00007F4363265266] ? (libc.so.6 + 0x2E266)
[0x00007F4363265312] ? (libc.so.6 + 0x2E312)
[0x0000559FD22C31C4] ZN21HttpClientTransaction15handleRedirectEv + 852 (splunkd + 0x11B31C4)
[0x0000559FD22C3932] ZN21HttpClientTransaction18finishTransactionEv + 354 (splunkd + 0x11B3932)
[0x0000559FD22C47C3] ZN20HttpClientConnection10parseReplyEPKcS1 + 1747 (splunkd + 0x11B47C3)
[0x0000559FD22C58C0] ZN20HttpClientConnection13dataAvailableEv + 160 (splunkd + 0x11B58C0)
[0x0000559FD2360B88] _ZN11TcpOutbound6
doioE18PollableDescriptor + 440 (splunkd + 0x1250B88)
[0x0000559FD2361C01] _ZN11TcpOutbound11when
eventsE18PollableDescriptor + 33 (splunkd + 0x1251C01)
[0x0000559FD22AA9B6] ZN8PolledFd8doeventEv + 134 (splunkd + 0x119A9B6)
[0x0000559FD22AB8BB] ZN9EventLoop3runEv + 651 (splunkd + 0x119B8BB)
[0x0000559FD23623B0] _ZN15TcpOutboundLoop3runEv + 16 (splunkd + 0x12523B0)
[0x0000559FD22C1712] _ZN21HttpClientTransaction22runSyncAndShutdownLoopEP15TcpOutboundLoop + 50 (splunkd + 0x11B1712)
[0x0000559FD22C17D5] _ZN21HttpClientTransaction7runSyncEv + 69 (splunkd + 0x11B17D5)
[0x0000559FD2235B41] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR28ApplicationUpdateTransaction + 145 (splunkd + 0x1125B41)
[0x0000559FD2235FB5] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR8Pathnameb + 357 (splunkd + 0x1125FB5)
[0x0000559FD20FF3AC] _ZN21LocalAppsAdminHandler13handleInstallER10ConfigInfoPK3StrS4
b + 2652 (splunkd + 0xFEF3AC)
[0x0000559FD20FF9ED] ZN21LocalAppsAdminHandler12handleCreateER10ConfigInfo + 253 (splunkd + 0xFEF9ED)
[0x0000559FD1DF980C] _ZN14MConfigHandler14executeHandlerER10ConfigInfo + 620 (splunkd + 0xCE980C)
[0x0000559FD1E09C7D] _ZN14MConfigHandler2goER10ConfigInfo + 189 (splunkd + 0xCF9C7D)
[0x0000559FD1E0A844] _ZN29AdminManagerReplyDataProvider2goEv + 804 (splunkd + 0xCFA844)
[0x0000559FD1EA3078] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0xD93078)
[0x0000559FD1E98B2F] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 31 (splunkd + 0xD88B2F)
[0x0000559FD22D9FE0] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 272 (splunkd + 0x11C9FE0)
[0x0000559FD19180B3] _ZN16TcpChannelThread4mainEv + 227 (splunkd + 0x8080B3)
[0x0000559FD2363440] _ZN6Thread8callMainEPv + 64 (splunkd + 0x1253440)
[0x00007F4363601E25] ? (libpthread.so.0 + 0x7E25)
[0x00007F436332F34D] clone + 109 (libc.so.6 + 0xF834D)
Linux / gssit-devops-qa / 3.10.0-693.1.1.el7.x86
64 / #1 SMP Thu Aug 3 08:15:31 EDT 2017 / x8664
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2017-10-05 16:38:25.291 -0400 Interrupt signal received
2017-10-05 16:40:18.548 -0400 splunkd started (build 4b804538c686)
2017-10-05 16:44:46.866 -0400 Interrupt signal received
2017-10-05 16:46:48.163 -0400 splunkd started (build 4b804538c686)
2017-10-05 22:55:50.030 -0400 Interrupt signal received
2017-10-05 22:57:42.807 -0400 splunkd started (build 4b804538c686)
splunkd: /home/build/build-src/kimono/src/util/HttpClientRequest.cpp:1860: void HttpClientTransaction::
handleRedirect(): Assertion _redirectReply == REPLY_EATING_NORMAL' failed.
2017-10-05 23:02:25.471 -0400 splunkd started (build 4b804538c686)
splunkd: /home/build/build-src/kimono/src/util/HttpClientRequest.cpp:1860: void HttpClientTransaction::_handleRedirect(): Assertion
redirectReply == REPLYEATING_NORMAL' failed.

Highlighted

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Splunk Employee
Splunk Employee

Hey @pyde88 - Can you let us know which apps specifically?

0 Karma
Highlighted

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Engager

I am getting this for any app. In particualr, I tried Github Addon and Monitoring Kubernetes etc.

0 Karma
Highlighted

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Communicator

So, how about install using .tar.gz file downloaded from splunkbase ?
You go Splunk->Manage apps->Install app from file and specify the path to the archive.

And check have you splunk user access to write data to splunk_home directory?

0 Karma
Highlighted

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Explorer

I'm receiving this same crash dump on 7.1.0 in a docker container.

I've set up my Proxy and RootCA settings to access the internet and download apps/content. Yes, we can certainly install from tar.gz packages downloaded from splunkbase, but that's not an appropriate answer to a hard crash like this.

Hopefully someone can investigate further.

0 Karma
Highlighted

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Path Finder

You could also download the .tar /.gz file directly from splunkbase.com and extract it to $SPLUNK_HOME/etc/apps
then restart Splunk.

0 Karma
Highlighted

Re: Splunk crashes when trying to install an app from "Browse more apps" section

Engager

Hello,

I realise this is an older question, but our Splunk search head just crashed with the exact same crash log when trying to install an app from 'browse more apps'. Also the setup is very similar: RHEL7, non root user, corporate proxy configured.

Did you ever manage to get this resolved?

Kr,

Jerome