Service fails when trying to install add-on

[AClerk]

Hi All,

Splunk enterprise version 8.1.3; Running on CENTOS 7.

I am trying to install an add-on from splunkbase.

Using Splunk web for that.

After I press the install button I get a fail error message.

When pressing the retry button, I get a message that the server is disconnected.

Checking the status -->

[root@SPLUNKSERVER bin]# ./splunk status
splunkd 117486 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.

Any idea how to troubleshoot this? And/Or resolve?

Cheers!

[richgalloway]

Check splunkd.log.  If you find a core file, open a support request with Splunk.

[AClerk]

The log contains these rows:

04-12-2021 10:43:52.692 +1000 WARN  LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
04-12-2021 10:43:52.694 +1000 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='/opt/splunk/var/log/splunk/splunkd.log'.
04-12-2021 10:44:08.284 +1000 INFO  DatabaseDirectoryManager - idx=_audit writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db' pendingBucketUpdates=0 innerLockTime=0.000. Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
04-12-2021 10:44:08.285 +1000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db duration=0.001
04-12-2021 10:44:08.286 +1000 INFO  DatabaseDirectoryManager - idx=_internal writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db' pendingBucketUpdates=0 innerLockTime=0.000. Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
04-12-2021 10:44:08.287 +1000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db duration=0.001
04-12-2021 10:44:08.293 +1000 INFO  DatabaseDirectoryManager - idx=_metrics writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_metrics/db' pendingBucketUpdates=0 innerLockTime=0.002. Reason='Buckets were rebuilt or tsidx-minified (bucket_count=5).'
04-12-2021 10:44:08.296 +1000 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_metrics/db duration=0.005
04-12-2021 10:44:27.977 +1000 WARN  LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
04-12-2021 10:44:28.417 +1000 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event (Mon Apr 12 10:43:06 2021). Context: source=/opt/splunk/var/log/splunk/splunkd_stderr.log|host=SPLUNKSERVER|splunkd_stderr|72

I updated limits.conf to

enable_install_apps = true

Now I see this WARN

04-12-2021 11:38:23.816 +1000 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event (Mon Apr 12 11:32:51 2021). Context: source=/opt/splunk/var/log/splunk/splunkd_stderr.log|host=SPLUNKSERVER|splunkd_stderr|72

I am still not able to install add-ons and the service is getting stopped/crashed.

 Cheers!

[AClerk]

In addition, a crash log is created. For example - crash-2021-04-12-11:38:23.log

Not sure what this means.

[build 63079c59e632] 2021-04-12 11:38:23
Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 51854 running under UID 1000.
 Crashing thread: TcpChannelThread
 Registers:
    RIP:  [0x00007F9904A403D7] gsignal + 55 (libc.so.6 + 0x363D7)
    RDI:  [0x000000000000CA8E]
    RSI:  [0x000000000000CC54]
    RBP:  [0x00007F9904B94E00]
    RSP:  [0x00007F98D9FF9ED8]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F990605B000]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0x0000000000000040]
    R9:  [0x00007F9904800080]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000206]
    R12:  [0x000055F795231F80]
    R13:  [0x000055F795232868]
    R14:  [0x00007F98D9FFA0D0]
    R15:  [0x00007F98D9FFA100]
    EFL:  [0x0000000000000206]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]

 OS: Linux
 Arch: x86-64

 Backtrace (PIC build):
  [0x00007F9904A403D7] gsignal + 55 (libc.so.6 + 0x363D7)
  [0x00007F9904A41AC8] abort + 328 (libc.so.6 + 0x37AC8)
  [0x00007F9904A391A6] ? (libc.so.6 + 0x2F1A6)
  [0x00007F9904A39252] ? (libc.so.6 + 0x2F252)
  [0x000055F793EA53A5] _ZN21HttpClientTransaction15_handleRedirectEv + 853 (splunkd + 0x22333A5)
  [0x000055F793EA5512] _ZN21HttpClientTransaction18_finishTransactionEv + 290 (splunkd + 0x2233512)
  [0x000055F793EA56A7] _ZN20HttpClientConnection10parseReplyEPKcS1_ + 199 (splunkd + 0x22336A7)
  [0x000055F793EA67F0] _ZN20HttpClientConnection13dataAvailableEv + 160 (splunkd + 0x22347F0)
  [0x000055F793F418D3] _ZN11TcpOutbound6_do_ioE18PollableDescriptor + 531 (splunkd + 0x22CF8D3)
  [0x000055F793F4286F] _ZN11TcpOutbound11when_eventsE18PollableDescriptor + 31 (splunkd + 0x22D086F)
  [0x000055F793E8A83C] _ZN8PolledFd8do_eventEv + 124 (splunkd + 0x221883C)
  [0x000055F793E8B750] _ZN9EventLoop3runEv + 624 (splunkd + 0x2219750)
  [0x000055F793F43022] _ZN15TcpOutboundLoop3runEv + 18 (splunkd + 0x22D1022)
  [0x000055F793EA1D0A] _ZN21HttpClientTransaction22runSyncAndShutdownLoopEP15TcpOutboundLoop + 106 (splunkd + 0x222FD0A)
  [0x000055F793EA1E24] _ZN21HttpClientTransaction7runSyncEv + 68 (splunkd + 0x222FE24)
  [0x000055F793DA7484] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR28ApplicationUpdateTransaction + 148 (splunkd + 0x2135484)
  [0x000055F793DA7811] _ZN18ApplicationUpdater20fetchUpdateFileByUriERK7FullUriRK3StrR8Pathnameb + 385 (splunkd + 0x2135811)
  [0x000055F793BC8CBA] _ZN21LocalAppsAdminHandler13handleInstallER10ConfigInfoPK3StrS4_bbb + 1818 (splunkd + 0x1F56CBA)
  [0x000055F793BC961E] _ZN21LocalAppsAdminHandler12handleCreateER10ConfigInfo + 414 (splunkd + 0x1F5761E)
  [0x000055F793826C5C] _ZN14MConfigHandler2goER10ConfigInfo + 188 (splunkd + 0x1BB4C5C)
  [0x000055F79382774E] _ZN29AdminManagerReplyDataProvider2goEv + 702 (splunkd + 0x1BB574E)
  [0x000055F793923E18] _ZN33ServicesEndpointReplyDataProvider9rawHandleEv + 88 (splunkd + 0x1CB1E18)
  [0x000055F79391727A] _ZN18RawRestHttpHandler10getPreBodyEP21HttpServerTransaction + 26 (splunkd + 0x1CA527A)
  [0x000055F793EBA470] _ZN32HttpThreadedCommunicationHandler11communicateER17TcpSyncDataBuffer + 224 (splunkd + 0x2248470)
  [0x000055F7931B4F37] _ZN16TcpChannelThread4mainEv + 343 (splunkd + 0x1542F37)
  [0x000055F793F44BE7] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22D2BE7)
  [0x00007F9904DDFEA5] ? (libpthread.so.0 + 0x7EA5)
  [0x00007F9904B089FD] clone + 109 (libc.so.6 + 0xFE9FD)
 Linux / SPLUNKSERVER / 3.10.0-957.el7.x86_64 / #1 SMP Thu Nov 8 23:39:32 UTC 2018 / x86_64
 Libc abort message: splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1934: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed.

 /etc/redhat-release: CentOS Linux release 7.9.2009 (Core)
 glibc version: 2.17
 glibc release: stable
Last errno: 0
Threads running: 75
Runtime: 332.324197s
argv: [splunkd -p 8089 start]
Regex JIT enabled

RE2 regex engine enabled

using CLOCK_MONOTONIC
Thread: "TcpChannelThread", did_join=0, ready_to_run=Y, main_thread=N, token=140294469175040
MutexByte: MutexByte-waiting={none}
commandForThread=0, nextIdle=(nil), requestAfterThread=0, _tpfd=0x7f98fce52000, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f98fcee11f0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil)
writeDataAvail @0x7f98fcee1258: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=(nil)
wbuf: ptr=0x7f98fcee1300, size=0x8000, rptr=0x0, wptr=0x0
HttpListeningConnection: _transactionActive=Y, _haveHadTransaction=Y, _alreadyLoggedTimeout=N
HttpTcpConnection: peer=10.150.11.43, _desiredCompressionLevel=6
RestHttpServerTransaction: _restPath="apps/local", namespaced=N, context=-/-, session=[user=admin, refcnt=2, touched=1618191503, refreshEligible=1618191658, removed=N, id=cc0d2972f698f829201bac617964ff8a, created=1618191181, refreshed=1618191433, expires=1618195033, initialLife=3600, createdBy=B0199C6F-4CCD-40C5-9BD1-6D118EDBCC19, ip=10.150.11.43, csrf=16653134604752637856]
HttpServerTransaction: _state=6, _shouldLog=Y, _startTime=1618191503.515650515054
REQUEST: POST /en-US/splunkd/__raw/services/apps/local HTTP/1.1
    Host: 10.150.120.30:8000
    Connection: keep-alive
    Content-Length: 154
    Accept: text/javascript, text/html, application/xml, text/xml, */*
    X-Requested-With: XMLHttpRequest
    X-Splunk-Form-Key: 16653134604752637856
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    Origin: http://10.150.120.30:8000
    Accept-Encoding: gzip, deflate
    Accept-Language: en-US,en;q=0.9
    Cookie: sessionid=qsnphp8s53wjayowkznio67oej8j6od7; splunkweb_csrf_token_8000=16653134604752637856; session_id_8000=e300df0ad1c9539c0e5aa553187f3b03401a446e; token_key=16653134604752637856; experience_id=3a86ff37-6359-aae8-f671-8e579d140089; splunkd_8000=dqAaDUFWtUvKSzzvaC2odNKHI6VnEbvzQo4N0y8cYAsRVT4UmUcio_LgyDnJ85NHq9VmVmpkpBrw2vkoi0nlc4AaRXGFFnmWpyK0xDdKGxIUzkxP1QYgV0H7LCOnVCP9QSCDrW7; csrftoken=nvZtjjqXEpKLO06tsW9m34ifnDYaCygHHiff9q0HiBbj0mmqPMKt6H105musZ65l
  _bytesReceived=154, _maximumRequestDataSize=524288000, _totalBytesExpectedOfRequestData=154
  _bytesLeftInRequestDataChunk=0, _requestTransferEncodingIsChunked=N, _receivingRequestDataForever=N
  _needToSetupRequestGunzip=N, _owedConsume=0, _wantSavedRequestData=N
  _100continue=0, _expectDisconnect=N, _overrideSourceState=0
POST arguments: {["filename"] = "true", ["auth"] = "qsnphp8s53wjayowkznio67oej8j6od7", ["name"] = "https://splunkbase.splunk.com/app/3103/release/2.1.0/download/"}
REPLY: 200 
    Set-Cookie: splunkd_8000=dqAaDUFWtUvKSzzvaC2odNKHI6VnEbvzQo4N0y8cYAsRVT4UmUcio_LgyDnJ85NHq9VmVmpkpBrw2vkoi0nlc4AaRXGFFnmWpyK0xDdKGxIUzkxP1QYgV0H7LCOnVCP9QSCDrW7; Path=/; HttpOnly; Max-Age=3600; Expires=Mon, 12 Apr 2021 02:38:23 GMT
    Set-Cookie: splunkweb_csrf_token_8000=16653134604752637856; Path=/; Max-Age=157680000; Expires=Sat, 11 Apr 2026 01:38:23 GMT
admin_handler="localapps"
MConfigHandler: name=localapps, _atomFormat=2, _customAction=
  caller args: id="https://splunkbase.splunk.com/app/3103/release/2.1.0/download/": { auth -> { _dataType=string _isMultiValue=Y, _values: ["qsnphp8s53wjayowkznio67oej8j6od7"] }, filename -> { _dataType=string _isMultiValue=Y, _values: ["true"] } }
  _docShowEntry=Y, _didFilter=N, _didPaginate=N
  _maxCount=30, posOffset=0, _requestedAction=1
  _shouldFilter=N, _shouldReload=N, _shouldAutoList=N, _sortSpecified=N
  _strict_mode=N, _list_new=N, _force_stanza_overwite=N, _force_app_context_on_write=Y
  sort keys: ["name"]
  sort modes: ["auto"]
  supported args: ["auth" type=0, "author" type=0, "configured" type=0, "deploy" type=0, "description" type=0, "explicit_appname" type=0, "filename" type=0, "label" type=0, "local_bundle" type=0, "name" type=0 (required), "session" type=0, "template" type=0, "update" type=0, "version" type=0, "visible" type=0]
  Paginator: offset=0, count=30
  _customStatusCode=0, _supportedActions=0x1EF, hasSession=Y
  _forceBoolNormalization=N  _forceRuleValidation=N, _contextMode=0, _didCapCheck=Y
  _ranSetup=Y, _restartRequired=N, _listingOne=N
  _userName=admin, _appName=search
ServicesEndpointReplyDataProvider: _setupState=0, _outputMode=2, _explicitOutputMode=Y
GET args: {}
  _allowedMethods={GET,POST,PUT,DELETE,HEAD,OPTIONS}, _preconditionState=0
  _wantsSeparateThread=N, _alreadyBuiltHeaders=N, _needToSendBody=Y
  _bodyBytesWritten=0, _chunkedState=0, _isLastTransaction=N
  _varyBy=0x8, _redirectUrl="", _downloadFilename="", _totalScheduledLength=0
  _willSendDataLater=N, _toSendState=0, _toSendSafe=Y
  _knowCompleteLength=N, _desiredCompressionLevel=6
  _replyContentEncoding= << identity, _cacheControl=0x0, _maxCacheSeconds=4294967295, _dontIncludeFrameOptions=N
In TcpChannel 0x7f98fce52000, _tcloop=0x7f990126a3c8, no async write data, _data._shouldKill=N, r/w_timeouts=5.000/300.000, timeout_count=0
SSL: inactive
rbuf: ptr=0x7f98fce520a0, size=0x2000, rptr=0x0, wptr=0x0
TcpChannelAcceptor: , tcloop=0x7f990126a3c8, _disabledReasons=0, _activeCount=1, _inflightSubordinateAccepts=0
HttpListener: ssl=N, _maxActiveConnections=1365, _wellBelowConnectionLimit=Y, _maxThreads=1365
SplunkdHttpListener: PORT: _allowGzip=Y, bind=http://:8000
  conf: _sslopt={rootCAPath="", caCertFile="", certFile="", privateKeyFile="", privateKeyPassword_set=N, commonNameToCheck="", altNameToCheck="", allowSslRenegotiation=Y, sslVersions="SSL3,TLS1.0,TLS1.1,TLS1.2", cipherSuite="", ecdhCurves="", useCompression==N, useSSL=2, quietShutdown==NdhFile="", shouldVerifyClientCert=N}, _allowSslRenegotiation=Y, _frameOptionsSameOrigin=Y, _strictTransportSecurityHeader=N, _includeSubDomains=N, _preload=N, _allowBasicAuth=N, _allowCookieAuth=N
  conf: _streamInWriteTimeout=5.000, _maxContentLength=524288000, _maxThreads=1365, _maxSockets=1365, _forceHttp10=0
_thread=0x7f98fcee10c0: commandForThread=0, nextIdle=(nil), requestAfterThread=0, _tpfd=0x7f98fce52000, writeCorkCount=0, terminateCallback=(nil), ioError=No error, lastError=No error, terminateError=No error
giveCmd @0x7f98fcee11f0: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=0, ok=Y, chan=(nil)
writeDataAvail @0x7f98fcee1258: _queuedOn=(nil), ran=N, wantWake=N, wantFailIfLoopDone=N, chan=(nil)
wbuf: ptr=0x7f98fcee1300, size=0x8000, rptr=0x0, wptr=0x0

First 512 bytes of PolledFd object @0x7f98e49fa140:
00000000  a8 8e db 95 f7 55 00 00  00 00 00 00 00 00 00 00  |.....U..........|
00000010  40 d7 e8 95 f7 55 00 00  00 00 00 00 00 00 00 00  |@....U..........|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  77 00 00 00 6f 00 00 00  |........w...o...|
00000040  c0 b6 9f f7 98 7f 00 00  1b 20 00 00 00 00 00 00  |......... ......|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  e8 d8 db 95 f7 55 00 00  00 00 00 00 00 00 00 00  |.....U..........|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  f8 a6 ff d9 98 7f 00 00  69 37 05 00 00 00 00 00  |........i7......|
00000090  f7 d0 f7 05 00 00 00 00  a0 86 01 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  40 a1 9f e4 98 7f 00 00  |........@.......|
000000b0  00 00 00 00 00 00 00 00  f0 a6 ff d9 98 7f 00 00  |................|
000000c0  c0 f3 30 01 99 7f 00 00  00 5c 9d e4 98 7f 00 00  |..0......\......|
000000d0  01 01 00 00 00 00 00 00  28 a2 9f e4 98 7f 00 00  |........(.......|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  ff ff ff ff 00 00 00 00  |................|
00000100  50 a2 9f e4 98 7f 00 00  00 00 00 00 00 00 00 00  |P...............|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  18 20 00 00 1b 20 00 00  e0 2e 00 00 00 00 00 00  |. ... ..........|
00000130  a0 86 01 00 00 00 00 00  a0 86 01 00 00 00 00 00  |................|
00000140  d0 07 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  c8 a2 9f e4 98 7f 00 00  00 00 00 00 00 00 00 00  |................|
00000190  90 f8 e8 95 f7 55 00 00  00 00 00 00 00 00 00 00  |.....U..........|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  f9 02 00 00 00 00 00 00  48 54 54 50 2f 31 2e 31  |........HTTP/1.1|
000001c0  20 33 30 32 20 46 6f 75  6e 64 0d 0a 43 6f 6e 74  | 302 Found..Cont|
000001d0  65 6e 74 2d 54 79 70 65  3a 20 74 65 78 74 2f 68  |ent-Type: text/h|
000001e0  74 6d 6c 3b 20 63 68 61  72 73 65 74 3d 75 74 66  |tml; charset=utf|
000001f0  2d 38 0d 0a 44 61 74 65  3a 20 4d 6f 6e 2c 20 31  |-8..Date: Mon, 1|
00000200
TcpClientConnection:  peer=10.150.30.87, port=8080In TcpOutputLoop 0x7f98e49fa140, _toloopp=0x7f98d9ffa6f0, _tstate=1, no async write data, isTerminated=N, destLoop=(nil), c/r/w/s timeouts=12.000/100.000/100.000/2.000, paused=0, timeout_count=0, ssl_shutdown_returned_zero=N
SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="ECDHE-RSA-AES128-GCM-SHA256", compression="none"

serr: No error, _wantEvents=8216, _setEvents=8219
rbuf: ptr=0x7f98e49fa2f8, size=0x4000, rptr=0x0, wptr=0x2F9
HttpClientConnection: _hc_state=12, _gunzip_initialized=N  _had_previous_transactions=Y, _can_reuse_connection=Y
ApplicationUpdateTransaction: file="/opt/splunk/var/run/72df7b140401f356.tar.gz", failureStr="", open=N
HttpClientTransaction: Connecting to host=http://10.150.30.87:8080
Request details: GET https://cdn.apps.splunk.com/media/private/8f49f3e2-e123-11ea-9bbf-02fa7700cf1c.tgz?response-content-disposition=attachment%3Bfilename%3D%22ta-for-nutanix-prism_210.tgz%22&Expires=1618193003&Signature=kRLSJ7w9FBXdrTX5lyU6bYgVwmW5RTg6EOtus5AO64nZJSLsdGv~JBHPeajfzSI1KK5GqcqFC8OdeRWMOgCMHKHLkfLo9A25NWeFk2uZoGL1fn-gRMbBNl4qL~JXfvctfiM5n84MvtR3hTkxHKAs2dzPyvW2PP8ppfj2ZgugEXnPU1bsdCF-TJ956NgyyltEqohpiKuAvwgKQlOfjtBMVbq8nM1S7Z8701Cv3J3hq0E8-meCck5fldILemKTQPheZ2VqPQ5djWTyFgfE-wwK0QJIPrIo-wtaTDTJdN225SE9hqPd3J2VMimBseJZ6z2cX2ANjRpsID1~YimdgUm3eA__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A
    X-Auth-Token: qsnphp8s53wjayowkznio67oej8j6od7
  _lastError=No error, _terminateEloopAfter=Y
  _connect_done=Y, _addrElem=0, _connectErrorPriority=0, _resolveError=""
  _useHttp11=Y, _allowTrailers=Y, _use_idle_connection=Y, _avoid_idle_connection_for_next_only=N, _last_on_connection=N, _send_content_type_even_if_no_body=N, _sniToSend=""
  _interpret_redirects=Y, _redirects_left=29, _redirectReply=2
  _doneSendingRequestData=N, _requestBytesExpected=0
  RESPONSE: HTTP/1.1 302 Found
    Content-Type: text/html; charset=utf-8
    Date: Mon, 12 Apr 2021 01:38:23 GMT
    Location: https://cdn.apps.splunk.com/media/private/8f49f3e2-e123-11ea-9bbf-02fa7700cf1c.tgz?response-content-disposition=attachment%3Bfilename%3D%22ta-for-nutanix-prism_210.tgz%22&Expires=1618193003&Signature=kRLSJ7w9FBXdrTX5lyU6bYgVwmW5RTg6EOtus5AO64nZJSLsdGv~JBHPeajfzSI1KK5GqcqFC8OdeRWMOgCMHKHLkfLo9A25NWeFk2uZoGL1fn-gRMbBNl4qL~JXfvctfiM5n84MvtR3hTkxHKAs2dzPyvW2PP8ppfj2ZgugEXnPU1bsdCF-TJ956NgyyltEqohpiKuAvwgKQlOfjtBMVbq8nM1S7Z8701Cv3J3hq0E8-meCck5fldILemKTQPheZ2VqPQ5djWTyFgfE-wwK0QJIPrIo-wtaTDTJdN225SE9hqPd3J2VMimBseJZ6z2cX2ANjRpsID1~YimdgUm3eA__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A
    Server: Apache
    Vary: Cookie
    Content-Length: 0
    Connection: keep-alive
  _bytesRx=0, _maybeCompressedBytesRx=0, _bytesExpected=0, _maxResponseSize=576460752303423487
  _acceptAndPass=identity, _acceptAndDecompress=identity, _activeDecompressPolicy=0, _remoteIndicatedCompression=identity
  _connectTimeout=10.000, _readTimeout=100.000, _writeTimeout=100.000
TcpClientConnectionPool: allowSsl=Y, _idleCount=0, _maxIdle=25, _addressOrder=0
  _sslShutdownTimeout=2.000, _idleTimeout=28.000, _idle_connection_trimmer_scheduled=N


x86 CPUID registers:
         0: 0000000D 756E6547 6C65746E 49656E69
         1: 00000F61 04020800 FFFA3203 1F8BFBFF
         2: 00000001 00000000 0000004D 002C307D
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000000 00000000 00000003 00000000
         6: 00000004 00000000 00000000 00000000
         7: 00000000 D09F4FBB 00000000 8C000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 00000000 00000000 00000000 00000000
         B: 00000000 00000000 00000000 00000000
         C: 00000000 00000000 00000000 00000000
         😧 00000000 00000000 00000000 00000000
  80000000: 80000008 756E6547 6C65746E 49656E69
  80000001: 00000F61 00000000 00000121 2C100800
  80000002: 65746E49 2952286C 6F655820 2952286E
  80000003: 6C695320 20726576 38303134 55504320
  80000004: 31204020 4730382E 00007A48 00000000
  80000005: 01FF01FF 01FF01FF 40020140 40020140
  80000006: 00000000 42004200 02008140 00808140
  80000007: 00000000 00000000 00000000 00000000
  80000008: 0000302E 00000000 00000001 00000000
terminating...

[richgalloway]

Report the crash to Splunk.  They will know how to interpret the log.

Neither of the warning messages cited should prevent Splunk from starting.  What did you see in splunkd_stderr?

[AClerk]

---

@richgalloway wrote:

What did you see in splunkd_stderr?

---

2021-04-12 10:35:29.548 +1000 splunkd started (build 63079c59e632) pid=49755
2021-04-12 10:38:00.792 +1000 Interrupt signal received sent by PID 50353, command="./splunk stop" (UID 1000, same as my group)
2021-04-12 10:38:56.510 +1000 splunkd started (build 63079c59e632) pid=50457
2021-04-12 10:41:42.554 +1000 Interrupt signal received sent by PID 51069, command="./splunk stop" (UID 1000, same as my group)
2021-04-12 10:43:06.695 +1000 splunkd started (build 63079c59e632) pid=51196
splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1934: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed.
2021-04-12 11:32:51.519 +1000 splunkd started (build 63079c59e632) pid=51854
splunkd: /opt/splunk/src/util/HttpClientRequest.cpp:1934: void HttpClientTransaction::_handleRedirect(): Assertion `_redirectReply == REPLY_EATING_NORMAL' failed.

---

@richgalloway wrote:

Report the crash to Splunk. 

---

How can I report? I don't have a support plan and using the free version.

Thanks!

[richgalloway]

You can't ask Splunk for support without a support plan.  Sorry.

Try removing the TA you added.  It could be preventing Splunk from starting.

[AClerk]

Thats the thing. I am not able to install from Splunk web.

I have no issues starting the service. It crashes when attempting to install add-on(s).

Thanks