Splunk service crashes when installing any add-on

[Hyperion]

Hi all,

I've installed the free 60-day Splunk Enterprise trial for testing purposes on a CentOS box, but am having encountering an issue where every time I try to install an add-on, the Splunk service crashes.  I am attempting to install through the browser, and this issue is not specific to any add-on (tried multiple different ones with the same result).

The crash log files don't seem to give any clues as to why this is occurring:

TcpClientConnection:  peer=172.17.1.69, port=8080In TcpOutputLoop 0x7f5eb6a5ab80, _toloopp=0x7f5eb73f26f0, _tstate=1, no async write data, isTerminated=N, destLoop=(nil), c/r/w/s timeouts=12.000/100.000/100.000/2.000, paused=0, timeout_count=0, ssl_shutdown_returned_zero=N
SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="ECDHE-RSA-AES128-GCM-SHA256", compression="none"

serr: No error, _wantEvents=8216, _setEvents=8219
rbuf: ptr=0x7f5eb6a5ad38, size=0x4000, rptr=0x0, wptr=0x313
HttpClientConnection: _hc_state=12, _gunzip_initialized=N  _had_previous_transactions=Y, _can_reuse_connection=Y
ApplicationUpdateTransaction: file="/opt/splunk/var/run/4024e94e0f412193.tar.gz", failureStr="", open=N
HttpClientTransaction: Connecting to host=http://172.17.1.69:8080
Request details: GET https://cdn.apps.splunk.com/media/private/51e661e4-edfc-11ea-8fee-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22microsoft-graph-security-api-add-on-for-splunk_121.tgz%22&Expires=1615870031&Signature=dPDDrPEHcebgKhSKS4SiX4BqntPkcvvJK1PAzosWRTkJUrf2JoRroh10sdTuFHZNcoDRi4qIqDFLT7WP6s29KZjDeEfe~tGrNeApUbggrienfdN49BjcVcsh0UXi1XPsYUXJaRAWb53jdHy13Qc856b8wRBYFESep8qMC~VADGGll4TPUROgIz5bHWEn0e~z8BycCGmOHSFdqssfmI9LIX2O7R6vkV5z-WD~HhjYOs~egPTn1knZkK0XuIzvOqUftBDXG6i070CpmBZ3XguRStHyFVqgZPn0B8QXdUBhIQBMYkHvZY62szK1NRFx6wolbrrkd73Hr0W5c~hrlY4QrA__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A
    X-Auth-Token: 9fdmh9gpu5z6bw2tekbu9k37sd62nn0h
  _lastError=No error, _terminateEloopAfter=Y
  _connect_done=Y, _addrElem=0, _connectErrorPriority=0, _resolveError=""
  _useHttp11=Y, _allowTrailers=Y, _use_idle_connection=Y, _avoid_idle_connection_for_next_only=N, _last_on_connection=N, _send_content_type_even_if_no_body=N, _sniToSend=""
  _interpret_redirects=Y, _redirects_left=29, _redirectReply=2
  _doneSendingRequestData=N, _requestBytesExpected=0
  RESPONSE: HTTP/1.1 302 Found
    Content-Type: text/html; charset=utf-8
    Date: Tue, 16 Mar 2021 04:22:11 GMT
    Location: https://cdn.apps.splunk.com/media/private/51e661e4-edfc-11ea-8fee-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22microsoft-graph-security-api-add-on-for-splunk_121.tgz%22&Expires=1615870031&Signature=dPDDrPEHcebgKhSKS4SiX4BqntPkcvvJK1PAzosWRTkJUrf2JoRroh10sdTuFHZNcoDRi4qIqDFLT7WP6s29KZjDeEfe~tGrNeApUbggrienfdN49BjcVcsh0UXi1XPsYUXJaRAWb53jdHy13Qc856b8wRBYFESep8qMC~VADGGll4TPUROgIz5bHWEn0e~z8BycCGmOHSFdqssfmI9LIX2O7R6vkV5z-WD~HhjYOs~egPTn1knZkK0XuIzvOqUftBDXG6i070CpmBZ3XguRStHyFVqgZPn0B8QXdUBhIQBMYkHvZY62szK1NRFx6wolbrrkd73Hr0W5c~hrlY4QrA__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A
    Server: Apache
    Vary: Cookie
    Content-Length: 0
    Connection: keep-alive
  _bytesRx=0, _maybeCompressedBytesRx=0, _bytesExpected=0, _maxResponseSize=576460752303423487
  _acceptAndPass=identity, _acceptAndDecompress=identity, _activeDecompressPolicy=0, _remoteIndicatedCompression=identity
  _connectTimeout=10.000, _readTimeout=100.000, _writeTimeout=100.000
TcpClientConnectionPool: allowSsl=Y, _idleCount=0, _maxIdle=25, _addressOrder=0
  _sslShutdownTimeout=2.000, _idleTimeout=28.000, _idle_connection_trimmer_scheduled=N


x86 CPUID registers:
         0: 0000000D 756E6547 6C65746E 49656E69
         1: 000206D2 04010800 9FBA2203 0F8BFBFF
         2: 76035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000000 00000000 00000000 00000000
         6: 00000004 00000000 00000000 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300401 0000007F 00000000 00000000
         B: 00000000 00000000 000000FD 00000004
         C: 00000000 00000000 00000000 00000000
         😧 00000000 00000000 00000000 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 6E6F6558 20295228 20555043 322D3545
  80000004: 20303836 20402030 30372E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 0000302B 00000000 00000000 00000000
terminating...

splunkd.log also doesn't show much:

03-16-2021 15:35:07.658 +1100 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_metrics/db duration=0.003
03-16-2021 15:35:08.092 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~49~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:08.093 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~50~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:08.309 +1100 INFO  ProcessTracker - (child_6__Fsck)  Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_metrics/db/db_1615869125_1615868908_51' took 128.5 milliseconds
03-16-2021 15:35:09.303 +1100 INFO  ProcessTracker - (child_7__Fsck)  Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_metrics/db/db_1615869125_1615868939_52' took 141.4 milliseconds
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Checking for localhost key pair
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
03-16-2021 15:35:09.760 +1100 INFO  KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
03-16-2021 15:35:10.072 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~51~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:10.160 +1100 WARN  ProcessTracker - (child_8__Fsck)  Fsck - Rebuilding entire bucket is not supported for "metric" bucket that has a "stubbed-out" rawdata journal. Only bloomfilter will be build
03-16-2021 15:35:10.160 +1100 INFO  ProcessTracker - (child_8__Fsck)  bloomfiltermaker - distinct_term_count failed: rc=-4
03-16-2021 15:35:10.160 +1100 WARN  ProcessTracker - (child_8__Fsck)  Fsck - Repair entire bucket, index=_metrics, tryWarmThenCold=1, bucket=/opt/splunk/var/lib/splunk/_metrics/db/db_1615535486_1615532541_4, exists=1, localrc=101, failReason=Bloomfilter rebuild for bkt='/opt/splunk/var/lib/splunk/_metrics/db/db_1615535486_1615532541_4' failed; rc=-4
03-16-2021 15:35:11.071 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~52~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:12.041 +1100 WARN  BucketMover - BucketManifestUpdateExitHandler: process handling bucket="db_1615535486_1615532541_4" exited with code=101; search for any previous messages that might have been produced by the external process
03-16-2021 15:35:12.041 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~4~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:18.590 +1100 WARN  LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
03-16-2021 15:35:31.579 +1100 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
03-16-2021 15:35:31.579 +1100 INFO  CascadingReplicationManager - Using value for property max_replication_threads=2.
03-16-2021 15:35:31.579 +1100 INFO  CascadingReplicationManager - Using value for property max_replication_jobs=5.
03-16-2021 15:35:34.964 +1100 INFO  MetricSchemaProcessor - channel confkey=source::/opt/splunk/var/log/splunk/metrics.log|host::AUSPS1SL0041|splunk_metrics_log|CLONE_CHANNEL has an event with no measure, will be skipped.
03-16-2021 15:35:49.966 +1100 WARN  LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
03-16-2021 15:35:50.236 +1100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event (Tue Mar 16 15:35:01 2021). Context: source=/opt/splunk/var/log/splunk/splunkd_stderr.log|host=AUSPS1SL0041|splunkd_stderr|72

Also ran a packet capture and found nothing out of the ordinary.

I'm aware that we can manually install add-ons from Splunkbase by extracting the .tar.gz but want to understand and solve the issue.  Anyone have any ideas?