Why is Splunkd not running after launching from an AMI image?

hantaliu — Fri, 09 Jun 2023 13:34:44 GMT

I am trying to launch a new instance from an image created by an existing EC2 instance that hosts Splunk. When I launch the new one, everything looks fine (Splunk was already installed, files remained unchanged, etc). However, I was not able to access Splunk app via <ipv4 address>:<port> (we are using 8443 instead but our inbound rule allows 8000, 8443, 8089...)

I checked the inbound rules and it is identical to the old one which have all correct ports setup.

splunkd 26175 was not running. Stopping splunk helpers... [ OK ] Done. Stopped helpers. Removing stale pid file... done. splunkd is not running. [FAILED] Splunk> The Notorious B.I.G. D.A.T.A. Checking prerequisites... Checking http port [8443]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _configtracker _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket boost_prod_connect history main summary Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-9.0.3-dd0128b1f8cd-linux-2.6-x86_64-manifest' File '/opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf' changed. Problems were found, please review your files and move customizations to local All preliminary checks passed. Starting splunk server daemon (splunkd)... PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security Done [ OK ] Waiting for web server at https://127.0.0.1:8443 to be available...................................splunkd 27894 was not running. Stopping splunk helpers... [ OK ] Done. Stopped helpers. Removing stale pid file... done. WARNING: web interface does not seem to be available!

Re: Splunkd not running after launching from an AMI image

PickleRick — Wed, 07 Jun 2023 19:57:46 GMT

This just shows that _something_ went wrong. We don't know what. Check your log - /opt/splunk/var/log/splunk/splunkd.log

Re: Splunkd not running after launching from an AMI image

hantaliu — Wed, 07 Jun 2023 20:05:01 GMT

Checked the log and it shows something wrong with the SSL setting?

06-07-2023 18:37:29.610 +0000 INFO DatabaseDirectoryManager [28341 indexerPipe] - idx=_audit writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='New hot bucket bid=_audit~47~5C52B298-3A3B-4A82-9F95-B9738E1D9BFB bucket_action=add' 06-07-2023 18:37:29.610 +0000 INFO DatabaseDirectoryManager [28341 indexerPipe] - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db duration=0.000 06-07-2023 18:37:29.619 +0000 INFO ServerRoles [28341 indexerPipe] - Declared role=indexer. 06-07-2023 18:37:30.122 +0000 WARN IntrospectionGenerator:resource_usage [28362 ExecProcessor] - SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security 06-07-2023 18:37:30.126 +0000 WARN IntrospectionGenerator:resource_usage [28362 ExecProcessor] - SSLCommon - PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security 06-07-2023 18:37:30.188 +0000 INFO ProcessTracker [27894 MainThread] - (child_0__Fsck) Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/audit/db/db_1686162521_1686162521_46' took 2703.9 milliseconds 06-07-2023 18:37:30.425 +0000 INFO TailingProcessor [28425 MainTailingThread] - TailWatcher initializing... 06-07-2023 18:37:30.425 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json. 06-07-2023 18:37:30.426 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk. 06-07-2023 18:37:30.426 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_hec. 06-07-2023 18:37:30.426 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/tracker.log*. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/introspection. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/python_upgrade_readiness_app. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/configuration_change.log. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log. 06-07-2023 18:37:30.427 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*. 06-07-2023 18:37:30.428 +0000 INFO TailReader [28425 MainTailingThread] - State transitioning from 1 to 0 (initOrResume). 06-07-2023 18:37:30.428 +0000 INFO TailReader [28425 MainTailingThread] - State transitioning from 1 to 0 (initOrResume). 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/etc/splunk.version. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/introspection. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/python_upgrade_readiness_app. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/splunk. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/watchdog. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/run/splunk/search_telemetry. 06-07-2023 18:37:30.428 +0000 INFO TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/spool/splunk. 06-07-2023 18:37:30.450 +0000 INFO TailReader [28443 tailreader0] - Registering metrics callback for: tailreader0 06-07-2023 18:37:30.450 +0000 INFO TailReader [28443 tailreader0] - Starting tailreader0 thread 06-07-2023 18:37:30.462 +0000 INFO TailReader [28444 batchreader0] - Registering metrics callback for: batchreader0 06-07-2023 18:37:30.462 +0000 INFO TailReader [28444 batchreader0] - Starting batchreader0 thread 06-07-2023 18:37:30.467 +0000 INFO ConfigWatcher [27902 HTTPDispatch] - Loaded configtracker settings with disabled=0 mode=auto log_throttling_disabled=1 log_throttling_threshold_ms=10.000 denylist= exclude_fields= 06-07-2023 18:37:30.529 +0000 WARN IntrospectionGenerator:resource_usage [28362 ExecProcessor] - SSLOptions - server.conf/[kvstore]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security 06-07-2023 18:37:30.643 +0000 INFO IntrospectionGenerator:resource_usage [28362 ExecProcessor] - RU_main - I-data gathering (Resource Usage) starting; period=10s 06-07-2023 18:37:30.733 +0000 INFO IntrospectionGenerator:resource_usage [28362 ExecProcessor] - RU_main - I-data gathering (IO Statistics) starting; interval=60s 06-07-2023 18:37:30.733 +0000 INFO IntrospectionGenerator:resource_usage [28362 ExecProcessor] - RU_main - Starting I-data gathering (IOWait Statistics). Interval_secs=10 06-07-2023 18:37:31.065 +0000 INFO ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - SplunkConfigChangeWatcher initializing... 06-07-2023 18:37:31.065 +0000 INFO ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - Kernel File Notification is enabled on this instance. inotify will be used for configuration tracking. 06-07-2023 18:37:31.067 +0000 INFO ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - Watching path: /opt/splunk/etc/system/local, /opt/splunk/etc/system/default, /opt/splunk/etc/apps, /opt/splunk/etc/users, /opt/splunk/etc/peer-apps, /opt/splunk/etc/instance.cfg 06-07-2023 18:37:31.195 +0000 INFO ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - Finding the deleted watched configuration files (while splunkd was down) completed in duration=0.127 secs 06-07-2023 18:37:31.362 +0000 INFO IndexerIf [28341 indexerPipe] - Asked to add or update bucket manifest values, bid=_audit~46~5C52B298-3A3B-4A82-9F95-B9738E1D9BFB 06-07-2023 18:37:31.438 +0000 INFO loader [27902 HTTPDispatch] - Limiting REST HTTP server to 21845 sockets 06-07-2023 18:37:31.438 +0000 INFO loader [27902 HTTPDispatch] - Limiting REST HTTP server to 161 threads 06-07-2023 18:37:31.438 +0000 WARN X509Verify [27902 HTTPDispatch] - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates> 06-07-2023 18:37:32.194 +0000 INFO UiHttpListener [28468 WebuiStartup] - Server supporting SSL versions TLS1.2 06-07-2023 18:37:32.194 +0000 INFO UiHttpListener [28468 WebuiStartup] - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 06-07-2023 18:37:32.194 +0000 INFO UiHttpListener [28468 WebuiStartup] - Using ECDH curves : prime256v1, secp384r1, secp521r1 06-07-2023 18:37:32.197 +0000 WARN X509Verify [28468 WebuiStartup] - X509 certificate (O=SplunkUser,CN=ip-172-31-46-102.us-west-2.compute.internal) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates> 06-07-2023 18:37:32.197 +0000 INFO UiHttpListener [28468 WebuiStartup] - Limiting UI HTTP server to 21845 sockets 06-07-2023 18:37:32.197 +0000 INFO UiHttpListener [28468 WebuiStartup] - Limiting UI HTTP server to 161 threads 06-07-2023 18:37:32.251 +0000 INFO DatabaseDirectoryManager [28321 IndexerService] - idx=_audit writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='IndexerService periodic manifest update' 06-07-2023 18:37:32.252 +0000 INFO DatabaseDirectoryManager [28321 IndexerService] - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db duration=0.001 06-07-2023 18:37:32.309 +0000 INFO ProxyConfig [28468 WebuiStartup] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled. 06-07-2023 18:37:32.310 +0000 INFO ProxyConfig [28468 WebuiStartup] - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled. 06-07-2023 18:37:32.310 +0000 INFO ProxyConfig [28468 WebuiStartup] - Failed to initialize the proxy_rules setting from server.conf for splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled. 06-07-2023 18:37:32.310 +0000 INFO ProxyConfig [28468 WebuiStartup] - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled. 06-07-2023 18:37:32.314 +0000 WARN SSLOptions [28468 WebuiStartup] - <internal>.conf/[<internal>]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security 06-07-2023 18:37:32.414 +0000 WARN SSLOptions [28468 WebuiStartup] - <internal>.conf/[<internal>]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security 06-07-2023 18:37:32.837 +0000 WARN SSLOptions [28394 SchedulerThread] - server.conf/[search_state]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security 06-07-2023 18:37:32.999 +0000 WARN ProcessTracker [27894 MainThread] - (child_1__Fsck) SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security 06-07-2023 18:37:34.574 +0000 INFO ExecProcessor [28362 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" splunk-dashboard-studio version is 1.7.3 06-07-2023 18:37:34.575 +0000 INFO ExecProcessor [28362 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" Content of /opt/splunk/etc/apps/splunk-dashboard-studio/kvstore_icon_status.conf is {'default': {'uploadedVersion': '1.7.3'}} 06-07-2023 18:37:34.575 +0000 INFO ExecProcessor [28362 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" Icons of splunk-dashboard-studio version 1.7.3 are already stored in kvstore collection. Skipping now and exiting.

Re: Splunkd not running after launching from an AMI image

isoutamo — Thu, 08 Jun 2023 21:19:06 GMT

i don’t think that those will explain, why splunkd doesn’t start. What else you have on logs after those?

r. Ismo

topic Re: Splunkd not running after launching from an AMI image in Splunk Enterprise

Why is Splunkd not running after launching from an AMI image?

Re: Splunkd not running after launching from an AMI image

Re: Splunkd not running after launching from an AMI image

Re: Splunkd not running after launching from an AMI image