topic Re: Splunk 8.0.2 and proxy conf in Installation

Splunk 8.0.2 and proxy conf

lies2yk — Wed, 08 Jul 2020 12:02:36 GMT

Hello

I'm having a strange problem on splunk 8 (clean install) and using the "proxyConfig" directive.

Inside the server.conf file I configured the following:
[ProxyConfig]
http_proxy = http://127.0.0.1:3128
https_proxy = https://127.0.0.1:3128
proxy_rules = *
no_proxy = localhost, 127.0.0.1, :: 1

Unfortunately, when I try to install an app via the web interface, the interface crashes immediately after I enter my credentials for downloading the app.
After a few seconds he throws me out and the login screen returns.

Please help me...

Thanks

Re: Splunk 8.0.2 and proxy conf

richgalloway — Wed, 08 Jul 2020 12:39:43 GMT

Have you checked splunkd.log?

Re: Splunk 8.0.2 and proxy conf

lies2yk — Wed, 08 Jul 2020 12:57:46 GMT

Here the log.
in this extraction you see Splunk 7.6 and proxy listen in port 3129 because i'm trying various release, but the same issue i've in 8.0.x and proxy configured on port 3128.

_ZN9EventLoop3runEv + 667 (splunkd + 0x181688B) [0x000055DE4DD40400] _ZN15TcpOutboundLoop3runEv + 16 (splunkd + 0x18DF400) [0x000055DE4DC8E8F7] _ZN21HttpClientTransaction22runSyncAndShutdownLoopEP15TcpOutboundLoop + 103 (splunkd + 0x182D8F7) [0x000055DE4DC8EA75] _ZN21HttpClientTransaction7runSyncEv + 69 (splunkd + 0x182DA75) [0x000055DE4DB855B1] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR28ApplicationUpdateTransaction + 145 (splunkd + 0x17245B1) [0x000055DE4DB85CF1] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR8Pathnameb + 385 (splunkd + 0x1724CF1) [0x000055DE4D9CD20E] _ZN21LocalAppsAdminHandler13handleInstallER10ConfigInfoPK3StrS4_bbb + 2174 (splunkd + 0x156C20E) [0x000055DE4D9CDD6A] _ZN21LocalAppsAdminHandler12handleCreateER10ConfigInfo + 426 (splunkd + 0x156CD6A) [0x000055DE4D64A52D] _ZN14MConfigHandler2goER10ConfigInfo + 189 (splunkd + 0x11E952D) [0x000055DE4D64B07F] _ZN29AdminManagerReplyDataProvider2goEv + 783 (splunkd + 0x11EA07F) [0x000055DE4D7273B8] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0x12C63B8) [0x000055DE4D71CEBF] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 31 (splunkd + 0x12BBEBF) [0x000055DE4DCA8390] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 272 (splunkd + 0x1847390) [0x000055DE4D057432] _ZN16TcpChannelThread4mainEv + 242 (splunkd + 0xBF6432) [0x000055DE4DD41638] _ZN6Thread8callMainEPv + 120 (splunkd + 0x18E0638) [0x00007FD2A54DAEA5] ? (libpthread.so.0 + 0x7EA5) [0x00007FD2A52038CD] clone + 109 (libc.so.6 + 0xFE8CD) Linux / ssplkshwsl01v / 3.10.0-1062.7.1.el7.x86_64 / #1 SMP Wed Nov 13 08:44:42 EST 2019 / x86_64 Last few lines of stderr (may contain info on assertion failure, but also could be old): 2020-07-08 13:39:16.950 +0200 splunkd started (build 47d8552a4d84) pid=47562 splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1895: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed. 2020-07-08 13:43:07.379 +0200 splunkd started (build 47d8552a4d84) pid=48059 splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1895: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed. 2020-07-08 13:46:35.317 +0200 splunkd started (build 47d8552a4d84) pid=48582 splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1895: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed. /etc/redhat-release: Red Hat Enterprise Linux Server release 7.6 (Maipo) glibc version: 2.17 glibc release: stable Last errno: 0 Threads running: 71 Runtime: 53.409999s argv: [splunkd -p 8089 restart] Regex JIT enabled using CLOCK_MONOTONIC Thread: "TcpChannelThread", did_join=0, ready_to_run=Y, main_thread=N First 8 bytes of Thread token @0x7fd2826ac510: 00000000 00 e7 1f 78 d2 7f 00 00 |...x....| 00000008 commandForThread=0, nextIdle=0x7fd2826a1200, requestAfterThread=0, _tpfd=0x7fd282622800, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error giveCmd @0x7fd2826ac680: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil) writeDataAvail @0x7fd2826ac6e0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=(nil) wbuf: ptr=0x7fd2826ac780, size=0x8000, rptr=0x0, wptr=0x0 HttpListeningConnection: _transactionActive=Y, _haveHadTransaction=Y, _alreadyLoggedTimeout=N HttpTcpConnection: peer=10.223.x.x, _desiredCompressionLevel=6 RestHttpServerTransaction: _restPath="apps/local", namespaced=N, context=-/-, session=[user=admin, refcnt=7, touched=1594208848, refreshEligible=1594209030, removed=N, id=9c9fcfeebea8f184466a7f3affad8eba, created=1594208805, expires=1594212405, initialLife=3600, createdBy=BFD1E27C-45EB-4B06-8A2C-330A2E2ABA35, ip=10.223.239.59, csrf=3758827135434845639] HttpServerTransaction: _state=6, _shouldLog=Y, _startTime=1594208848.524956575153 REQUEST: POST /it-IT/splunkd/__raw/services/apps/local HTTP/1.1 Host: 10.187.80.196:8000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0 Accept: text/javascript, text/html, application/xml, text/xml, */* Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3 Accept-Encoding: gzip, deflate X-Splunk-Form-Key: 3758827135434845639 Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Requested-With: XMLHttpRequest Content-Length: 154 Origin: http://10.x.x.x:8000 Connection: keep-alive Cookie: session_id_8000=5ee84200ab939d8fcd6be5586971ec00ee6f8bcf; splunkweb_csrf_token_8000=3758827135434845639; splunkd_8000=FSoqtVC0ZhAOn3Pt8kcBvhOaOm0Z9D8Gji5M5^TZSup_8EuvGKgseg4Qdso7Vw4XQ_Q3^yd2eHeJczDPZIaM_C_32bhM8nuewdP22HyZMlY9qW4Mp_XfQVTk0X8e; csrftoken=4QQp5WpLEnIkk2Vw6UxiWQOMoCuXZs3Vrk4SOKB3lmCLuLf4Ks4JEZ770LBSApmX; sessionid=pm4b2jwciiwt2zx1e8o2r11h1pvr6eka _bytesReceived=154, _maximumRequestDataSize=524288000, _totalBytesExpectedOfRequestData=154 _bytesLeftInRequestDataChunk=0, _requestTransferEncodingIsChunked=N, _receivingRequestDataForever=N _needToSetupRequestGunzip=N, _owedConsume=0, _wantSavedRequestData=N _100continue=0, _expectDisconnect=N, _overrideSourceState=0 POST arguments: {["filename"] = "true", ["auth"] = "pm4b2jwciiwt2zx1e8o2r11h1pvr6eka", ["name"] = "https://splunkbase.splunk.com/app/4335/release/3.2.1/download/"} REPLY: 200 Set-Cookie: splunkd_8000=FSoqtVC0ZhAOn3Pt8kcBvhOaOm0Z9D8Gji5M5^TZSup_8EuvGKgseg4Qdso7Vw4XQ_Q3^yd2eHeJczDPZIaM_C_32bhM8nuewdP22HyZMlY9qW4Mp_XfQVTk0X8e; Path=/; HttpOnly; Max-Age=3600; Expires=Wed, 08 Jul 2020 12:47:28 GMT Set-Cookie: splunkweb_csrf_token_8000=3758827135434845639; Path=/; Max-Age=157680000; Expires=Mon, 07 Jul 2025 11:47:28 GMT admin_handler="localapps" MConfigHandler: name=localapps, _atomFormat=2, _customAction= caller args: id="https://splunkbase.splunk.com/app/4335/release/3.2.1/download/": { auth -> { _dataType=string _isMultiValue=Y, _values: ["pm4b2jwciiwt2zx1e8o2r11h1pvr6eka"] }, filename -> { _dataType=string _isMultiValue=Y, _values: ["true"] } } _docShowEntry=Y, _didFilter=N, _didPaginate=N _maxCount=30, posOffset=0, _requestedAction=1 _shouldFilter=N, _shouldReload=N, _shouldAutoList=N, _sortSpecified=N _strict_mode=N, _list_new=N, _force_stanza_overwite=N, _force_app_context_on_write=Y sort keys: ["name"] sort modes: ["auto"] supported args: ["auth" type=0, "author" type=0, "configured" type=0, "deploy" type=0, "description" type=0, "explicit_appname" type=0, "filename" type=0, "label" type=0, "local_bundle" type=0, "name" type=0 (required), "session" type=0, "template" type=0, "update" type=0, "version" type=0, "visible" type=0] Paginator: offset=0, count=30 _customStatusCode=0, _supportedActions=0x1EF, hasSession=Y _forceBoolNormalization=N _forceRuleValidation=N, _contextMode=0, _didCapCheck=Y _ranSetup=Y, _restartRequired=N, _listingOne=N _userName=admin, _appName=search ServicesEndpointReplyDataProvider: _setupState=0, _outputMode=2, _explicitOutputMode=Y GET args: {} _allowedMethods={GET,POST,PUT,DELETE,HEAD,OPTIONS}, _preconditionState=0 _wantsSeparateThread=N, _alreadyBuiltHeaders=N, _needToSendBody=Y _bodyBytesWritten=0, _chunkedState=0, _isLastTransaction=N _varyBy=0x8, _redirectUrl="", _downloadFilename="", _totalScheduledLength=0 _willSendDataLater=N, _toSendState=0, _toSendSafe=Y _knowCompleteLength=N, _desiredCompressionLevel=6 _replyContentEncoding= << identity, _cacheControl=0x0, _maxCacheSeconds=4294967295, _dontIncludeFrameOptions=N In TcpChannel 0x7fd282622800, _tcloop=0x7fd28a8ae2f8, no async write data, _data._shouldKill=N, r/w_timeouts=5.000/300.000, timeout_count=0 SSL: inactive rbuf: ptr=0x7fd2826228a0, size=0x2000, rptr=0x0, wptr=0x0 TcpChannelAcceptor: , tcloop=0x7fd28a8ae2f8, _disabledReasons=0, _activeCount=6, _inflightSubordinateAccepts=0 HttpListener: ssl=N, _maxActiveConnections=1365, _wellBelowConnectionLimit=Y, _maxThreads=1365 SplunkdHttpListener: PORT: _allowGzip=Y, bind=http://:8000 conf: _sslopt={rootCAPath="", caCertFile="", certFile="", privateKeyFile="", privateKeyPassword_set=N, commonNameToCheck="", altNameToCheck="", allowSslRenegotiation=Y, sslVersions="SSL3,TLS1.0,TLS1.1,TLS1.2", cipherSuite="", ecdhCurves="", useCompression==N, useSSL=2, quietShutdown==NdhFile="", shouldVerifyClientCert=N}, _allowSslRenegotiation=Y, _frameOptionsSameOrigin=Y, _strictTransportSecurityHeader=N, _allowBasicAuth=N, _allowCookieAuth=N conf: _streamInWriteTimeout=5.000, _maxContentLength=524288000, _maxThreads=1365, _maxSockets=1365, _forceHttp10=0 _thread=0x7fd2826ac500: commandForThread=0, nextIdle=0x7fd2826a1200, requestAfterThread=0, _tpfd=0x7fd282622800, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error giveCmd @0x7fd2826ac680: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil) writeDataAvail @0x7fd2826ac6e0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=(nil) wbuf: ptr=0x7fd2826ac780, size=0x8000, rptr=0x0, wptr=0x0 First 512 bytes of PolledFd object @0x7fd2a1b05c40: 00000000 08 27 5c 4f de 55 00 00 00 00 00 00 00 00 00 00 |.'\O.U..........| 00000010 20 a2 67 4f de 55 00 00 00 00 00 00 00 00 00 00 | .gO.U..........| 00000020 e0 73 ba a1 d2 7f 00 00 00 00 00 00 00 00 00 00 |.s..............| 00000030 1e 00 00 00 00 00 00 00 5e 00 00 00 61 00 00 00 |........^...a...| 00000040 80 25 bc a1 d2 7f 00 00 1b 20 00 00 d2 7f 00 00 |.%....... ......| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000060 c8 71 5c 4f de 55 00 00 00 00 00 00 00 00 00 00 |.q\O.U..........| 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000080 68 c6 1f 78 d2 7f 00 00 77 d4 8c 00 00 00 00 00 |h..x....w.......| 00000090 26 20 c2 3a 00 00 00 00 a0 86 01 00 00 00 00 00 |& .:............| 000000a0 40 5c b0 a1 d2 7f 00 00 00 00 00 00 d2 7f 00 00 |@\..............| 000000b0 60 c6 1f 78 d2 7f 00 00 80 a8 8a 8a d2 7f 00 00 |`..x............| 000000c0 00 77 ba a1 d2 7f 00 00 01 01 00 00 d2 7f 00 00 |.w..............| 000000d0 20 5d b0 a1 d2 7f 00 00 00 00 00 00 00 00 00 00 | ]..............| 000000e0 00 00 00 00 00 00 00 00 00 01 b0 a1 d2 7f 00 00 |................| 000000f0 ff ff ff ff d2 7f 00 00 48 5d b0 a1 d2 7f 00 00 |........H]......| 00000100 00 00 00 00 00 00 00 00 00 89 86 8a d2 7f 00 00 |................| 00000110 60 5d b0 a1 d2 7f 00 00 18 20 00 00 1b 20 00 00 |`]....... ... ..| 00000120 e0 2e 00 00 00 00 00 00 a0 86 01 00 00 00 00 00 |................| 00000130 a0 86 01 00 00 00 00 00 d0 07 00 00 00 00 00 00 |................| 00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000150 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................| 00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000170 00 00 00 00 00 00 00 00 c0 5d b0 a1 d2 7f 00 00 |.........]......| 00000180 00 00 00 00 00 00 00 00 30 c2 67 4f de 55 00 00 |........0.gO.U..| 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000001a0 00 00 00 00 00 00 00 00 f1 02 00 00 00 00 00 00 |................| 000001b0 48 54 54 50 2f 31 2e 31 20 33 30 32 20 46 6f 75 |HTTP/1.1 302 Fou| 000001c0 6e 64 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 |nd..Content-Type| 000001d0 3a 20 74 65 78 74 2f 68 74 6d 6c 3b 20 63 68 61 |: text/html; cha| 000001e0 72 73 65 74 3d 75 74 66 2d 38 0d 0a 44 61 74 65 |rset=utf-8..Date| 000001f0 3a 20 57 65 64 2c 20 30 38 20 4a 75 6c 20 32 30 |: Wed, 08 Jul 20| 00000200 TcpClientConnection: peer=127.0.0.1, port=3129In TcpOutputLoop 0x7fd2a1b05c40, _toloopp=0x7fd2781fc660, _tstate=1, no async write data, isTerminated=N, destLoop=(nil), c/r/w/s timeouts=12.000/100.000/100.000/2.000, paused=0, timeout_count=0, ssl_shutdown_returned_zero=N SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="ECDHE-RSA-AES128-GCM-SHA256", compression="none" serr: No error, _wantEvents=8216, _setEvents=8219 rbuf: ptr=0x7fd2a1b05df0, size=0x4000, rptr=0x0, wptr=0x2F1 HttpClientConnection: _hc_state=12, _gunzip_initialized=N _had_previous_transactions=Y, _can_reuse_connection=Y ApplicationUpdateTransaction: file="/opt/splunk/var/run/98f4c71e0bb74726.tar.gz", failureStr="", open=N HttpClientTransaction: Connecting to host=http://127.0.0.1:3129 Request details: GET https://cdn.apps.splunk.com/media/private/4461d6f0-ad99-11ea-b41b-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22misp42splunk_321.tgz%22&Expires=1594210348&Signature=Ui3RerZf5bPFUMmx2nWPhPo1c8FWt4u7M-BxHaEFYySybOhTRMlgbdLcElE03VN8B18qw97j16XbT~YbcrSR7Cup69HFk53~sORx1Th9~4bnaAJ4defPvIJ-31CMHkjy6MwFA6QWsBf~obVtNOtn6zzT2eY7E48fPMnmqk1KuljlUS04HK40Yz5QGOpHgekM96SFdezdOp3cJV9wcxcYztIWwKNEdOQZFnlXfmvqfM0gywIIDvtoj4Y2Knj~pXrku83hf3igDNEWdZXzXqSkqodQbkclQ-6tYn9GObcK5QJt6sDpxu4trYwM-eBEYdQcgaR98PQbWUK-swW07UThIg__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A X-Auth-Token: pm4b2jwciiwt2zx1e8o2r11h1pvr6eka _lastError=No error, _terminateEloopAfter=Y _connect_done=Y, _addrElem=0, _connectErrorPriority=0, _resolveError="" _useHttp11=Y, _allowTrailers=Y, _use_idle_connection=Y, _avoid_idle_connection_for_next_only=N, _last_on_connection=N, _send_content_type_even_if_no_body=N, _sniToSend="" _interpret_redirects=Y, _redirects_left=29, _redirectReply=2 _doneSendingRequestData=N, _requestBytesExpected=0 RESPONSE: HTTP/1.1 302 Found Content-Type: text/html; charset=utf-8 Date: Wed, 08 Jul 2020 11:47:28 GMT Location: https://cdn.apps.splunk.com/media/private/4461d6f0-ad99-11ea-b41b-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22misp42splunk_321.tgz%22&Expires=1594210348&Signature=Ui3RerZf5bPFUMmx2nWPhPo1c8FWt4u7M-BxHaEFYySybOhTRMlgbdLcElE03VN8B18qw97j16XbT~YbcrSR7Cup69HFk53~sORx1Th9~4bnaAJ4defPvIJ-31CMHkjy6MwFA6QWsBf~obVtNOtn6zzT2eY7E48fPMnmqk1KuljlUS04HK40Yz5QGOpHgekM96SFdezdOp3cJV9wcxcYztIWwKNEdOQZFnlXfmvqfM0gywIIDvtoj4Y2Knj~pXrku83hf3igDNEWdZXzXqSkqodQbkclQ-6tYn9GObcK5QJt6sDpxu4trYwM-eBEYdQcgaR98PQbWUK-swW07UThIg__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A Server: Apache Vary: Cookie Content-Length: 0 Connection: keep-alive _bytesRx=0, _maybeCompressedBytesRx=0, _bytesExpected=0, _maxResponseSize=576460752303423487 _acceptAndPass=identity, _acceptAndDecompress=identity, _activeDecompressPolicy=0, _remoteIndicatedCompression=identity _connectTimeout=10.000, _readTimeout=100.000, _writeTimeout=100.000 TcpClientConnectionPool: allowSsl=Y, _idleCount=0, _maxIdle=25, _addressOrder=0 _sslShutdownTimeout=2.000, _idleTimeout=28.000, _idle_connection_trimmer_scheduled=N x86 CPUID registers: 0: 0000000D 756E6547 6C65746E 49656E69 1: 00050654 1E010800 FEFA3203 0FABFBFF 2: 76036301 00F0B5FF 00000000 00C30000 3: 00000000 00000000 00000000 00000000 4: 00000000 00000000 00000000 00000000 5: 00000000 00000000 00000000 00000000 6: 00000004 00000000 00000000 00000000 7: 00000000 00000000 00000000 00000000 8: 00000000 00000000 00000000 00000000 9: 00000000 00000000 00000000 00000000 A: 07300401 0000007F 00000000 00000000 B: 00000000 00000000 000000FD 0000001E C: 00000000 00000000 00000000 00000000 😧 00000000 00000000 00000000 00000000 80000000: 80000008 00000000 00000000 00000000 80000001: 00000000 00000000 00000101 2C100800 80000002: 65746E49 2952286C 6F655820 2952286E 80000003: 6C6F4720 31352064 43203831 40205550 80000004: 332E3220 7A484730 00000000 00000000 80000005: 00000000 00000000 00000000 00000000 80000006: 00000000 00000000 01006040 00000000 80000007: 00000000 00000000 00000000 00000100 80000008: 00003028 00000000 00000000 00000000 terminating...

07-08-2020 13:46:36.009 +0200 INFO ProxyConfig - Successfully initialized https_proxy=http://127.0.0.1:3129 from server.conf for splunkd. 07-08-2020 13:46:36.009 +0200 INFO ProxyConfig - Successfully initialized http_proxy=http://127.0.0.1:3129 from server.conf for splunkd.

Re: Splunk 8.0.2 and proxy conf

lies2yk — Wed, 08 Jul 2020 13:48:43 GMT

And here the complete crash log: Sorry for long text....!

[build 47d8552a4d84] 2020-07-08 15:41:18 Received fatal signal 6 (Aborted). Cause: Signal sent by PID 54177 running under UID 0. Crashing thread: TcpChannelThread Registers: RIP: [0x00007FD784903377] gsignal + 55 (libc.so.6 + 0x36377) RDI: [0x000000000000D3A1] RSI: [0x000000000000D4A7] RBP: [0x00007FD784A57C20] RSP: [0x00007FD758DFBE28] RAX: [0x0000000000000000] RBX: [0x00007FD785FE6000] RCX: [0xFFFFFFFFFFFFFFFF] RDX: [0x0000000000000006] R8: [0x00000000000000A0] R9: [0xFEFEFEFEFF092D63] R10: [0x0000000000000008] R11: [0x0000000000000202] R12: [0x0000556F90E44920] R13: [0x0000556F90F6A3C0] R14: [0x00007FD758DFC050] R15: [0x00007FD786038E08] EFL: [0x0000000000000202] TRAPNO: [0x0000000000000000] ERR: [0x0000000000000000] CSGSFS: [0xFFFF000000000033] OLDMASK: [0x0000000000000000] OS: Linux Arch: x86-64 Backtrace (PIC build): [0x00007FD784903377] gsignal + 55 (libc.so.6 + 0x36377) [0x00007FD784904A68] abort + 328 (libc.so.6 + 0x37A68) [0x00007FD7848FC196] ? (libc.so.6 + 0x2F196) [0x00007FD7848FC242] ? (libc.so.6 + 0x2F242) [0x0000556F8FDD2684] _ZN21HttpClientTransaction15_handleRedirectEv + 852 (splunkd + 0x182F684) [0x0000556F8FDD3572] _ZN21HttpClientTransaction18_finishTransactionEv + 274 (splunkd + 0x1830572) [0x0000556F8FDD476B] _ZN20HttpClientConnection10parseReplyEPKcS1_ + 1691 (splunkd + 0x183176B) [0x0000556F8FDD5860] _ZN20HttpClientConnection13dataAvailableEv + 160 (splunkd + 0x1832860) [0x0000556F8FE80C77] _ZN11TcpOutbound6_do_ioE18PollableDescriptor + 423 (splunkd + 0x18DDC77) [0x0000556F8FE81C51] _ZN11TcpOutbound11when_eventsE18PollableDescriptor + 33 (splunkd + 0x18DEC51) [0x0000556F8FDB88B6] _ZN8PolledFd8do_eventEv + 134 (splunkd + 0x18158B6) [0x0000556F8FDB988B] _ZN9EventLoop3runEv + 667 (splunkd + 0x181688B) [0x0000556F8FE82400] _ZN15TcpOutboundLoop3runEv + 16 (splunkd + 0x18DF400) [0x0000556F8FDD08F7] _ZN21HttpClientTransaction22runSyncAndShutdownLoopEP15TcpOutboundLoop + 103 (splunkd + 0x182D8F7) [0x0000556F8FDD0A75] _ZN21HttpClientTransaction7runSyncEv + 69 (splunkd + 0x182DA75) [0x0000556F8FCC75B1] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR28ApplicationUpdateTransaction + 145 (splunkd + 0x17245B1) [0x0000556F8FCC7CF1] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR8Pathnameb + 385 (splunkd + 0x1724CF1) [0x0000556F8FB0F20E] _ZN21LocalAppsAdminHandler13handleInstallER10ConfigInfoPK3StrS4_bbb + 2174 (splunkd + 0x156C20E) [0x0000556F8FB0FD6A] _ZN21LocalAppsAdminHandler12handleCreateER10ConfigInfo + 426 (splunkd + 0x156CD6A) [0x0000556F8F78C52D] _ZN14MConfigHandler2goER10ConfigInfo + 189 (splunkd + 0x11E952D) [0x0000556F8F78D07F] _ZN29AdminManagerReplyDataProvider2goEv + 783 (splunkd + 0x11EA07F) [0x0000556F8F8693B8] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0x12C63B8) [0x0000556F8F85EEBF] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 31 (splunkd + 0x12BBEBF) [0x0000556F8FDEA390] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 272 (splunkd + 0x1847390) [0x0000556F8F199432] _ZN16TcpChannelThread4mainEv + 242 (splunkd + 0xBF6432) [0x0000556F8FE83638] _ZN6Thread8callMainEPv + 120 (splunkd + 0x18E0638) [0x00007FD784CA2EA5] ? (libpthread.so.0 + 0x7EA5) [0x00007FD7849CB8CD] clone + 109 (libc.so.6 + 0xFE8CD) Linux / ssplkshwsl01v / 3.10.0-1062.7.1.el7.x86_64 / #1 SMP Wed Nov 13 08:44:42 EST 2019 / x86_64 Last few lines of stderr (may contain info on assertion failure, but also could be old): 2020-07-08 15:37:44.361 +0200 Interrupt signal received sent by PID 52646 (UID 0) 2020-07-08 15:38:02.990 +0200 splunkd started (build 47d8552a4d84) pid=52734 2020-07-08 15:38:37.273 +0200 Interrupt signal received sent by PID 53122 (UID 0) 2020-07-08 15:38:55.899 +0200 splunkd started (build 47d8552a4d84) pid=53213 2020-07-08 15:39:41.400 +0200 Interrupt signal received sent by PID 53607 (UID 0) 2020-07-08 15:39:59.024 +0200 splunkd started (build 47d8552a4d84) pid=53694 2020-07-08 15:40:29.952 +0200 Interrupt signal received sent by PID 54019 (UID 0) 2020-07-08 15:40:46.579 +0200 splunkd started (build 47d8552a4d84) pid=54177 splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1895: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed. /etc/redhat-release: Red Hat Enterprise Linux Server release 7.6 (Maipo) glibc version: 2.17 glibc release: stable Last errno: 0 Threads running: 72 Runtime: 32.336326s argv: [splunkd -p 8089 restart] Regex JIT enabled using CLOCK_MONOTONIC Thread: "TcpChannelThread", did_join=0, ready_to_run=Y, main_thread=N First 8 bytes of Thread token @0x7fd764aa7c10: 00000000 00 e7 df 58 d7 7f 00 00 |...X....| 00000008 commandForThread=0, nextIdle=0x7fd764a9ca40, requestAfterThread=0, _tpfd=0x7fd764a22800, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error giveCmd @0x7fd764aa7d80: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil) writeDataAvail @0x7fd764aa7de0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=0x7fd764a27800 wbuf: ptr=0x7fd764aa7e80, size=0x8000, rptr=0x0, wptr=0x0 HttpListeningConnection: _transactionActive=Y, _haveHadTransaction=Y, _alreadyLoggedTimeout=N HttpTcpConnection: peer=10.22.x.x, _desiredCompressionLevel=6 RestHttpServerTransaction: _restPath="apps/local", namespaced=N, context=-/-, session=[user=admin, refcnt=7, touched=1594215678, refreshEligible=1594215881, removed=N, id=d385f669fbf4984aa1cb560528dfaa30, created=1594215656, expires=1594219256, initialLife=3600, createdBy=BFD1E27C-45EB-4B06-8A2C-330A2E2ABA35, ip=10.22.x.x, csrf=10733806047818363417] HttpServerTransaction: _state=6, _shouldLog=Y, _startTime=1594215678.515153535249 REQUEST: POST /it-IT/splunkd/__raw/services/apps/local HTTP/1.1 Host: 10.x.x.x:8000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0 Accept: text/javascript, text/html, application/xml, text/xml, */* Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3 Accept-Encoding: gzip, deflate X-Splunk-Form-Key: 10733806047818363417 Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Requested-With: XMLHttpRequest Content-Length: 154 Origin: http://10.x.x.x:8000 Connection: keep-alive Cookie: session_id_8000=a9c42a41b5e763284dac4c49d4b06841b1d82666; splunkweb_csrf_token_8000=10733806047818363417; splunkd_8000=5QQJr40SASizD5Al8z600hxQLcvPoDcT1hdAK8WOYqoGS2fxjtXzRwbZc4bLMrQRY6EcJjHeLVOvdGeFwnQVaH1Dy^kFIoQM1mLhG74CGNKFmNcYJzRnBkIVI7FvPUm3fC; csrftoken=fvvvLb5LgzUUPaulTpMMAo0bIHZDZUHFHhBRJLv7ouWAnZA0TKxJZdtNMv5MX26K; sessionid=pm4b2jwciiwt2zx1e8o2r11h1pvr6eka _bytesReceived=154, _maximumRequestDataSize=524288000, _totalBytesExpectedOfRequestData=154 _bytesLeftInRequestDataChunk=0, _requestTransferEncodingIsChunked=N, _receivingRequestDataForever=N _needToSetupRequestGunzip=N, _owedConsume=0, _wantSavedRequestData=N _100continue=0, _expectDisconnect=N, _overrideSourceState=0 POST arguments: {["filename"] = "true", ["auth"] = "pm4b2jwciiwt2zx1e8o2r11h1pvr6eka", ["name"] = "https://splunkbase.splunk.com/app/4335/release/3.2.1/download/"} REPLY: 200 Set-Cookie: splunkd_8000=5QQJr40SASizD5Al8z600hxQLcvPoDcT1hdAK8WOYqoGS2fxjtXzRwbZc4bLMrQRY6EcJjHeLVOvdGeFwnQVaH1Dy^kFIoQM1mLhG74CGNKFmNcYJzRnBkIVI7FvPUm3fC; Path=/; HttpOnly; Max-Age=3600; Expires=Wed, 08 Jul 2020 14:41:18 GMT Set-Cookie: splunkweb_csrf_token_8000=10733806047818363417; Path=/; Max-Age=157680000; Expires=Mon, 07 Jul 2025 13:41:18 GMT admin_handler="localapps" MConfigHandler: name=localapps, _atomFormat=2, _customAction= caller args: id="https://splunkbase.splunk.com/app/4335/release/3.2.1/download/": { auth -> { _dataType=string _isMultiValue=Y, _values: ["pm4b2jwciiwt2zx1e8o2r11h1pvr6eka"] }, filename -> { _dataType=string _isMultiValue=Y, _values: ["true"] } } _docShowEntry=Y, _didFilter=N, _didPaginate=N _maxCount=30, posOffset=0, _requestedAction=1 _shouldFilter=N, _shouldReload=N, _shouldAutoList=N, _sortSpecified=N _strict_mode=N, _list_new=N, _force_stanza_overwite=N, _force_app_context_on_write=Y sort keys: ["name"] sort modes: ["auto"] supported args: ["auth" type=0, "author" type=0, "configured" type=0, "deploy" type=0, "description" type=0, "explicit_appname" type=0, "filename" type=0, "label" type=0, "local_bundle" type=0, "name" type=0 (required), "session" type=0, "template" type=0, "update" type=0, "version" type=0, "visible" type=0] Paginator: offset=0, count=30 _customStatusCode=0, _supportedActions=0x1EF, hasSession=Y _forceBoolNormalization=N _forceRuleValidation=N, _contextMode=0, _didCapCheck=Y _ranSetup=Y, _restartRequired=N, _listingOne=N _userName=admin, _appName=search ServicesEndpointReplyDataProvider: _setupState=0, _outputMode=2, _explicitOutputMode=Y GET args: {} _allowedMethods={GET,POST,PUT,DELETE,HEAD,OPTIONS}, _preconditionState=0 _wantsSeparateThread=N, _alreadyBuiltHeaders=N, _needToSendBody=Y _bodyBytesWritten=0, _chunkedState=0, _isLastTransaction=N _varyBy=0x8, _redirectUrl="", _downloadFilename="", _totalScheduledLength=0 _willSendDataLater=N, _toSendState=0, _toSendSafe=Y _knowCompleteLength=N, _desiredCompressionLevel=6 _replyContentEncoding= << identity, _cacheControl=0x0, _maxCacheSeconds=4294967295, _dontIncludeFrameOptions=N In TcpChannel 0x7fd764a22800, _tcloop=0x7fd769ea82f8, no async write data, _data._shouldKill=N, r/w_timeouts=5.000/300.000, timeout_count=0 SSL: inactive rbuf: ptr=0x7fd764a228a0, size=0x2000, rptr=0x0, wptr=0x0 TcpChannelAcceptor: , tcloop=0x7fd769ea82f8, _disabledReasons=0, _activeCount=6, _inflightSubordinateAccepts=0 HttpListener: ssl=N, _maxActiveConnections=1365, _wellBelowConnectionLimit=Y, _maxThreads=1365 SplunkdHttpListener: PORT: _allowGzip=Y, bind=http://:8000 conf: _sslopt={rootCAPath="", caCertFile="", certFile="", privateKeyFile="", privateKeyPassword_set=N, commonNameToCheck="", altNameToCheck="", allowSslRenegotiation=Y, sslVersions="SSL3,TLS1.0,TLS1.1,TLS1.2", cipherSuite="", ecdhCurves="", useCompression==N, useSSL=2, quietShutdown==NdhFile="", shouldVerifyClientCert=N}, _allowSslRenegotiation=Y, _frameOptionsSameOrigin=Y, _strictTransportSecurityHeader=N, _allowBasicAuth=N, _allowCookieAuth=N conf: _streamInWriteTimeout=5.000, _maxContentLength=524288000, _maxThreads=1365, _maxSockets=1365, _forceHttp10=0 _thread=0x7fd764aa7c00: commandForThread=0, nextIdle=0x7fd764a9ca40, requestAfterThread=0, _tpfd=0x7fd764a22800, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error giveCmd @0x7fd764aa7d80: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil) writeDataAvail @0x7fd764aa7de0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=0x7fd764a27800 wbuf: ptr=0x7fd764aa7e80, size=0x8000, rptr=0x0, wptr=0x0 First 512 bytes of PolledFd object @0x7fd781abc580: 00000000 08 47 70 91 6f 55 00 00 00 00 00 00 00 00 00 00 |.Gp.oU..........| 00000010 20 c2 7b 91 6f 55 00 00 00 00 00 00 00 00 00 00 | .{.oU..........| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 00 00 00 00 00 00 00 00 95 00 00 00 40 00 00 00 |............@...| 00000040 40 1a ab 81 d7 7f 00 00 1b 20 00 00 00 00 00 00 |@........ ......| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000060 c8 91 70 91 6f 55 00 00 00 00 00 00 00 00 00 00 |..p.oU..........| 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000080 68 c6 df 58 d7 7f 00 00 25 ef 8c 00 00 00 00 00 |h..X....%.......| 00000090 06 7b c8 35 00 00 00 00 a0 86 01 00 00 00 00 00 |.{.5............| 000000a0 80 c5 ab 81 d7 7f 00 00 00 00 00 00 00 00 00 00 |................| 000000b0 60 c6 df 58 d7 7f 00 00 80 78 ea 69 d7 7f 00 00 |`..X.....x.i....| 000000c0 00 d0 a5 81 d7 7f 00 00 01 01 00 00 00 00 00 00 |................| 000000d0 60 c6 ab 81 d7 7f 00 00 00 00 00 00 00 00 00 00 |`...............| 000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000f0 ff ff ff ff 00 00 00 00 88 c6 ab 81 d7 7f 00 00 |................| 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000110 00 00 00 00 00 00 00 00 18 20 00 00 1b 20 00 00 |......... ... ..| 00000120 e0 2e 00 00 00 00 00 00 a0 86 01 00 00 00 00 00 |................| 00000130 a0 86 01 00 00 00 00 00 d0 07 00 00 00 00 00 00 |................| 00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000150 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................| 00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000170 00 00 00 00 00 00 00 00 00 c7 ab 81 d7 7f 00 00 |................| 00000180 00 00 00 00 00 00 00 00 30 e2 7b 91 6f 55 00 00 |........0.{.oU..| 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000001a0 00 00 00 00 00 00 00 00 f1 02 00 00 00 00 00 00 |................| 000001b0 48 54 54 50 2f 31 2e 31 20 33 30 32 20 46 6f 75 |HTTP/1.1 302 Fou| 000001c0 6e 64 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 |nd..Content-Type| 000001d0 3a 20 74 65 78 74 2f 68 74 6d 6c 3b 20 63 68 61 |: text/html; cha| 000001e0 72 73 65 74 3d 75 74 66 2d 38 0d 0a 44 61 74 65 |rset=utf-8..Date| 000001f0 3a 20 57 65 64 2c 20 30 38 20 4a 75 6c 20 32 30 |: Wed, 08 Jul 20| 00000200 TcpClientConnection: peer=127.0.0.1, port=3128In TcpOutputLoop 0x7fd781abc580, _toloopp=0x7fd758dfc660, _tstate=1, no async write data, isTerminated=N, destLoop=(nil), c/r/w/s timeouts=12.000/100.000/100.000/2.000, paused=0, timeout_count=0, ssl_shutdown_returned_zero=N SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="ECDHE-RSA-AES128-GCM-SHA256", compression="none" serr: No error, _wantEvents=8216, _setEvents=8219 rbuf: ptr=0x7fd781abc730, size=0x4000, rptr=0x0, wptr=0x2F1 HttpClientConnection: _hc_state=12, _gunzip_initialized=N _had_previous_transactions=Y, _can_reuse_connection=Y ApplicationUpdateTransaction: file="/opt/splunk/var/run/9e609a5b8abf3c36.tar.gz", failureStr="", open=N HttpClientTransaction: Connecting to host=http://127.0.0.1:3128 Request details: GET https://cdn.apps.splunk.com/media/private/4461d6f0-ad99-11ea-b41b-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22misp42splunk_321.tgz%22&Expires=1594217178&Signature=E-fAq-mlh5a1h5vxCyFLx39m2e6k1w54a3C4bY58UcXeagrZU-A2rakFGsVCjbmFbeX-NtHqfa1~Gv7Pdh27eX9QZrp6alHVxd1E9QnhbF3C9aEZAPlaeCbMzXCoPPFsyJt5H1l1joux0lSTU5tcr3VLPtAtYO5bcn6B5vD3lOvjvQ0zX-0WyXhofkCXTi4SMUOeXse51hvwX16DB4BVoYp8sdJ28yqmoao2-BIZPxLbOI6VNUE275T13jPcNFPqhRTRwFEKEqgSCqopfoeddR3gFqO5ngoEJufA622HcoZnwyFmbkkQn9CUAAQORUKDPaz7MXIHoPBoTlopdXAPbw__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A X-Auth-Token: pm4b2jwciiwt2zx1e8o2r11h1pvr6eka _lastError=No error, _terminateEloopAfter=Y _connect_done=Y, _addrElem=0, _connectErrorPriority=0, _resolveError="" _useHttp11=Y, _allowTrailers=Y, _use_idle_connection=Y, _avoid_idle_connection_for_next_only=N, _last_on_connection=N, _send_content_type_even_if_no_body=N, _sniToSend="" _interpret_redirects=Y, _redirects_left=29, _redirectReply=2 _doneSendingRequestData=N, _requestBytesExpected=0 RESPONSE: HTTP/1.1 302 Found Content-Type: text/html; charset=utf-8 Date: Wed, 08 Jul 2020 13:41:18 GMT Location: https://cdn.apps.splunk.com/media/private/4461d6f0-ad99-11ea-b41b-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22misp42splunk_321.tgz%22&Expires=1594217178&Signature=E-fAq-mlh5a1h5vxCyFLx39m2e6k1w54a3C4bY58UcXeagrZU-A2rakFGsVCjbmFbeX-NtHqfa1~Gv7Pdh27eX9QZrp6alHVxd1E9QnhbF3C9aEZAPlaeCbMzXCoPPFsyJt5H1l1joux0lSTU5tcr3VLPtAtYO5bcn6B5vD3lOvjvQ0zX-0WyXhofkCXTi4SMUOeXse51hvwX16DB4BVoYp8sdJ28yqmoao2-BIZPxLbOI6VNUE275T13jPcNFPqhRTRwFEKEqgSCqopfoeddR3gFqO5ngoEJufA622HcoZnwyFmbkkQn9CUAAQORUKDPaz7MXIHoPBoTlopdXAPbw__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A Server: Apache Vary: Cookie Content-Length: 0 Connection: keep-alive _bytesRx=0, _maybeCompressedBytesRx=0, _bytesExpected=0, _maxResponseSize=576460752303423487 _acceptAndPass=identity, _acceptAndDecompress=identity, _activeDecompressPolicy=0, _remoteIndicatedCompression=identity _connectTimeout=10.000, _readTimeout=100.000, _writeTimeout=100.000 TcpClientConnectionPool: allowSsl=Y, _idleCount=2, _maxIdle=25, _addressOrder=0 _sslShutdownTimeout=2.000, _idleTimeout=28.000, _idle_connection_trimmer_scheduled=N x86 CPUID registers: 0: 0000000D 756E6547 6C65746E 49656E69 1: 00050654 10010800 FEFA3203 0FABFBFF 2: 76036301 00F0B5FF 00000000 00C30000 3: 00000000 00000000 00000000 00000000 4: 00000000 00000000 00000000 00000000 5: 00000000 00000000 00000000 00000000 6: 00000004 00000000 00000000 00000000 7: 00000000 00000000 00000000 00000000 8: 00000000 00000000 00000000 00000000 9: 00000000 00000000 00000000 00000000 A: 07300401 0000007F 00000000 00000000 B: 00000000 00000000 000000FD 00000010 C: 00000000 00000000 00000000 00000000 😧 00000000 00000000 00000000 00000000 80000000: 80000008 00000000 00000000 00000000 80000001: 00000000 00000000 00000101 2C100800 80000002: 65746E49 2952286C 6F655820 2952286E 80000003: 6C6F4720 31352064 43203831 40205550 80000004: 332E3220 7A484730 00000000 00000000 80000005: 00000000 00000000 00000000 00000000 80000006: 00000000 00000000 01006040 00000000 80000007: 00000000 00000000 00000000 00000100 80000008: 00003028 00000000 00000000 00000000 terminating...