All Apps and Add-ons

Splunk service crashes when installing any add-on

Hyperion
Observer

Hi all,

I've installed the free 60-day Splunk Enterprise trial for testing purposes on a CentOS box, but am having encountering an issue where every time I try to install an add-on, the Splunk service crashes.  I am attempting to install through the browser, and this issue is not specific to any add-on (tried multiple different ones with the same result).

The crash log files don't seem to give any clues as to why this is occurring:

 

 

TcpClientConnection:  peer=172.17.1.69, port=8080In TcpOutputLoop 0x7f5eb6a5ab80, _toloopp=0x7f5eb73f26f0, _tstate=1, no async write data, isTerminated=N, destLoop=(nil), c/r/w/s timeouts=12.000/100.000/100.000/2.000, paused=0, timeout_count=0, ssl_shutdown_returned_zero=N
SSL: version="TLSv1.2", state="SSL negotiation finished successfully", cipher="ECDHE-RSA-AES128-GCM-SHA256", compression="none"

serr: No error, _wantEvents=8216, _setEvents=8219
rbuf: ptr=0x7f5eb6a5ad38, size=0x4000, rptr=0x0, wptr=0x313
HttpClientConnection: _hc_state=12, _gunzip_initialized=N  _had_previous_transactions=Y, _can_reuse_connection=Y
ApplicationUpdateTransaction: file="/opt/splunk/var/run/4024e94e0f412193.tar.gz", failureStr="", open=N
HttpClientTransaction: Connecting to host=http://172.17.1.69:8080
Request details: GET https://cdn.apps.splunk.com/media/private/51e661e4-edfc-11ea-8fee-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22microsoft-graph-security-api-add-on-for-splunk_121.tgz%22&Expires=1615870031&Signature=dPDDrPEHcebgKhSKS4SiX4BqntPkcvvJK1PAzosWRTkJUrf2JoRroh10sdTuFHZNcoDRi4qIqDFLT7WP6s29KZjDeEfe~tGrNeApUbggrienfdN49BjcVcsh0UXi1XPsYUXJaRAWb53jdHy13Qc856b8wRBYFESep8qMC~VADGGll4TPUROgIz5bHWEn0e~z8BycCGmOHSFdqssfmI9LIX2O7R6vkV5z-WD~HhjYOs~egPTn1knZkK0XuIzvOqUftBDXG6i070CpmBZ3XguRStHyFVqgZPn0B8QXdUBhIQBMYkHvZY62szK1NRFx6wolbrrkd73Hr0W5c~hrlY4QrA__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A
    X-Auth-Token: 9fdmh9gpu5z6bw2tekbu9k37sd62nn0h
  _lastError=No error, _terminateEloopAfter=Y
  _connect_done=Y, _addrElem=0, _connectErrorPriority=0, _resolveError=""
  _useHttp11=Y, _allowTrailers=Y, _use_idle_connection=Y, _avoid_idle_connection_for_next_only=N, _last_on_connection=N, _send_content_type_even_if_no_body=N, _sniToSend=""
  _interpret_redirects=Y, _redirects_left=29, _redirectReply=2
  _doneSendingRequestData=N, _requestBytesExpected=0
  RESPONSE: HTTP/1.1 302 Found
    Content-Type: text/html; charset=utf-8
    Date: Tue, 16 Mar 2021 04:22:11 GMT
    Location: https://cdn.apps.splunk.com/media/private/51e661e4-edfc-11ea-8fee-06add55d78f8.tgz?response-content-disposition=attachment%3Bfilename%3D%22microsoft-graph-security-api-add-on-for-splunk_121.tgz%22&Expires=1615870031&Signature=dPDDrPEHcebgKhSKS4SiX4BqntPkcvvJK1PAzosWRTkJUrf2JoRroh10sdTuFHZNcoDRi4qIqDFLT7WP6s29KZjDeEfe~tGrNeApUbggrienfdN49BjcVcsh0UXi1XPsYUXJaRAWb53jdHy13Qc856b8wRBYFESep8qMC~VADGGll4TPUROgIz5bHWEn0e~z8BycCGmOHSFdqssfmI9LIX2O7R6vkV5z-WD~HhjYOs~egPTn1knZkK0XuIzvOqUftBDXG6i070CpmBZ3XguRStHyFVqgZPn0B8QXdUBhIQBMYkHvZY62szK1NRFx6wolbrrkd73Hr0W5c~hrlY4QrA__&Key-Pair-Id=APKAISM7Q7KZPNKOIT7A
    Server: Apache
    Vary: Cookie
    Content-Length: 0
    Connection: keep-alive
  _bytesRx=0, _maybeCompressedBytesRx=0, _bytesExpected=0, _maxResponseSize=576460752303423487
  _acceptAndPass=identity, _acceptAndDecompress=identity, _activeDecompressPolicy=0, _remoteIndicatedCompression=identity
  _connectTimeout=10.000, _readTimeout=100.000, _writeTimeout=100.000
TcpClientConnectionPool: allowSsl=Y, _idleCount=0, _maxIdle=25, _addressOrder=0
  _sslShutdownTimeout=2.000, _idleTimeout=28.000, _idle_connection_trimmer_scheduled=N


x86 CPUID registers:
         0: 0000000D 756E6547 6C65746E 49656E69
         1: 000206D2 04010800 9FBA2203 0F8BFBFF
         2: 76035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000000 00000000 00000000 00000000
         6: 00000004 00000000 00000000 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300401 0000007F 00000000 00000000
         B: 00000000 00000000 000000FD 00000004
         C: 00000000 00000000 00000000 00000000
         😧 00000000 00000000 00000000 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 6E6F6558 20295228 20555043 322D3545
  80000004: 20303836 20402030 30372E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 0000302B 00000000 00000000 00000000
terminating...

 

 

splunkd.log also doesn't show much:

 

03-16-2021 15:35:07.658 +1100 INFO  DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_metrics/db duration=0.003
03-16-2021 15:35:08.092 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~49~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:08.093 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~50~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:08.309 +1100 INFO  ProcessTracker - (child_6__Fsck)  Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_metrics/db/db_1615869125_1615868908_51' took 128.5 milliseconds
03-16-2021 15:35:09.303 +1100 INFO  ProcessTracker - (child_7__Fsck)  Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/_metrics/db/db_1615869125_1615868939_52' took 141.4 milliseconds
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Checking for localhost key pair
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Public key already exists: /opt/splunk/etc/auth/distServerKeys/trusted.pem
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Finished reading public key for localhost: /opt/splunk/etc/auth/distServerKeys/trusted.pem
03-16-2021 15:35:09.759 +1100 INFO  KeyManagerLocalhost - Reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
03-16-2021 15:35:09.760 +1100 INFO  KeyManagerLocalhost - Finished reading private key for localhost: /opt/splunk/etc/auth/distServerKeys/private.pem
03-16-2021 15:35:10.072 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~51~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:10.160 +1100 WARN  ProcessTracker - (child_8__Fsck)  Fsck - Rebuilding entire bucket is not supported for "metric" bucket that has a "stubbed-out" rawdata journal. Only bloomfilter will be build
03-16-2021 15:35:10.160 +1100 INFO  ProcessTracker - (child_8__Fsck)  bloomfiltermaker - distinct_term_count failed: rc=-4
03-16-2021 15:35:10.160 +1100 WARN  ProcessTracker - (child_8__Fsck)  Fsck - Repair entire bucket, index=_metrics, tryWarmThenCold=1, bucket=/opt/splunk/var/lib/splunk/_metrics/db/db_1615535486_1615532541_4, exists=1, localrc=101, failReason=Bloomfilter rebuild for bkt='/opt/splunk/var/lib/splunk/_metrics/db/db_1615535486_1615532541_4' failed; rc=-4
03-16-2021 15:35:11.071 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~52~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:12.041 +1100 WARN  BucketMover - BucketManifestUpdateExitHandler: process handling bucket="db_1615535486_1615532541_4" exited with code=101; search for any previous messages that might have been produced by the external process
03-16-2021 15:35:12.041 +1100 INFO  IndexerIf - Asked to add or update bucket manifest values, bid=_metrics~4~97934EC8-853C-4F5B-A25A-A2F76DB6E0FB
03-16-2021 15:35:18.590 +1100 WARN  LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
03-16-2021 15:35:31.579 +1100 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
03-16-2021 15:35:31.579 +1100 INFO  CascadingReplicationManager - Using value for property max_replication_threads=2.
03-16-2021 15:35:31.579 +1100 INFO  CascadingReplicationManager - Using value for property max_replication_jobs=5.
03-16-2021 15:35:34.964 +1100 INFO  MetricSchemaProcessor - channel confkey=source::/opt/splunk/var/log/splunk/metrics.log|host::AUSPS1SL0041|splunk_metrics_log|CLONE_CHANNEL has an event with no measure, will be skipped.
03-16-2021 15:35:49.966 +1100 WARN  LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
03-16-2021 15:35:50.236 +1100 WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event (Tue Mar 16 15:35:01 2021). Context: source=/opt/splunk/var/log/splunk/splunkd_stderr.log|host=AUSPS1SL0041|splunkd_stderr|72

 

Also ran a packet capture and found nothing out of the ordinary.

I'm aware that we can manually install add-ons from Splunkbase by extracting the .tar.gz but want to understand and solve the issue.  Anyone have any ideas?

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...