Why is Splunkd not running after launching from an AMI image?

[hantaliu]

I am trying to launch a new instance from an image created by an existing EC2 instance that hosts Splunk. When I launch the new one, everything looks fine (Splunk was already installed, files remained unchanged, etc). However, I was not able to access Splunk app via <ipv4 address>:<port> (we are using 8443 instead but our inbound rule allows 8000, 8443, 8089...) 

I checked the inbound rules and it is identical to the old one which have all correct ports setup. 

splunkd 26175 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.
splunkd is not running.                                    [FAILED]

Splunk> The Notorious B.I.G. D.A.T.A.

Checking prerequisites...
        Checking http port [8443]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration... Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _configtracker _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket boost_prod_connect history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunk/splunk-9.0.3-dd0128b1f8cd-linux-2.6-x86_64-manifest'
File '/opt/splunk/etc/apps/splunk_instrumentation/default/savedsearches.conf' changed.
        Problems were found, please review your files and move customizations to local
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
                                                           [  OK  ]

Waiting for web server at https://127.0.0.1:8443 to be available...................................splunkd 27894 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... done.


WARNING: web interface does not seem to be available!

[hantaliu]

Checked the log and it shows something wrong with the SSL setting?

06-07-2023 18:37:29.610 +0000 INFO  DatabaseDirectoryManager [28341 indexerPipe] - idx=_audit writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='New hot bucket bid=_audit~47~5C52B298-3A3B-4A82-9F95-B9738E1D9BFB bucket_action=add'
06-07-2023 18:37:29.610 +0000 INFO  DatabaseDirectoryManager [28341 indexerPipe] - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db duration=0.000
06-07-2023 18:37:29.619 +0000 INFO  ServerRoles [28341 indexerPipe] - Declared role=indexer.
06-07-2023 18:37:30.122 +0000 WARN  IntrospectionGenerator:resource_usage [28362 ExecProcessor] -   SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
06-07-2023 18:37:30.126 +0000 WARN  IntrospectionGenerator:resource_usage [28362 ExecProcessor] -   SSLCommon - PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
06-07-2023 18:37:30.188 +0000 INFO  ProcessTracker [27894 MainThread] - (child_0__Fsck)  Fsck - (entire bucket) Rebuild for bucket='/opt/splunk/var/lib/splunk/audit/db/db_1686162521_1686162521_46' took 2703.9 milliseconds
06-07-2023 18:37:30.425 +0000 INFO  TailingProcessor [28425 MainTailingThread] - TailWatcher initializing...
06-07-2023 18:37:30.425 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json.
06-07-2023 18:37:30.426 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
06-07-2023 18:37:30.426 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_hec.
06-07-2023 18:37:30.426 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/tracker.log*.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/introspection.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/python_upgrade_readiness_app.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/configuration_change.log.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/license_usage_summary.log.
06-07-2023 18:37:30.427 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/watchdog/watchdog.log*.
06-07-2023 18:37:30.428 +0000 INFO  TailReader [28425 MainTailingThread] - State transitioning from 1 to 0 (initOrResume).
06-07-2023 18:37:30.428 +0000 INFO  TailReader [28425 MainTailingThread] - State transitioning from 1 to 0 (initOrResume).
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/etc/splunk.version.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/introspection.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/python_upgrade_readiness_app.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/splunk.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/log/watchdog.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/run/splunk/search_telemetry.
06-07-2023 18:37:30.428 +0000 INFO  TailingProcessor [28425 MainTailingThread] - Adding watch on path: /opt/splunk/var/spool/splunk.
06-07-2023 18:37:30.450 +0000 INFO  TailReader [28443 tailreader0] - Registering metrics callback for: tailreader0
06-07-2023 18:37:30.450 +0000 INFO  TailReader [28443 tailreader0] - Starting tailreader0 thread
06-07-2023 18:37:30.462 +0000 INFO  TailReader [28444 batchreader0] - Registering metrics callback for: batchreader0
06-07-2023 18:37:30.462 +0000 INFO  TailReader [28444 batchreader0] - Starting batchreader0 thread
06-07-2023 18:37:30.467 +0000 INFO  ConfigWatcher [27902 HTTPDispatch] - Loaded configtracker settings with disabled=0 mode=auto log_throttling_disabled=1 log_throttling_threshold_ms=10.000 denylist= exclude_fields=
06-07-2023 18:37:30.529 +0000 WARN  IntrospectionGenerator:resource_usage [28362 ExecProcessor] -   SSLOptions - server.conf/[kvstore]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
06-07-2023 18:37:30.643 +0000 INFO  IntrospectionGenerator:resource_usage [28362 ExecProcessor] -   RU_main - I-data gathering (Resource Usage) starting; period=10s
06-07-2023 18:37:30.733 +0000 INFO  IntrospectionGenerator:resource_usage [28362 ExecProcessor] -   RU_main - I-data gathering (IO Statistics) starting; interval=60s
06-07-2023 18:37:30.733 +0000 INFO  IntrospectionGenerator:resource_usage [28362 ExecProcessor] -   RU_main - Starting I-data gathering (IOWait Statistics). Interval_secs=10
06-07-2023 18:37:31.065 +0000 INFO  ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - SplunkConfigChangeWatcher initializing...
06-07-2023 18:37:31.065 +0000 INFO  ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - Kernel File Notification is enabled on this instance. inotify will be used for configuration tracking.
06-07-2023 18:37:31.067 +0000 INFO  ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - Watching path: /opt/splunk/etc/system/local, /opt/splunk/etc/system/default, /opt/splunk/etc/apps, /opt/splunk/etc/users, /opt/splunk/etc/peer-apps, /opt/splunk/etc/instance.cfg
06-07-2023 18:37:31.195 +0000 INFO  ConfigWatcher [28445 SplunkConfigChangeWatcherThread] - Finding the deleted watched configuration files (while splunkd was down) completed in duration=0.127 secs
06-07-2023 18:37:31.362 +0000 INFO  IndexerIf [28341 indexerPipe] - Asked to add or update bucket manifest values, bid=_audit~46~5C52B298-3A3B-4A82-9F95-B9738E1D9BFB
06-07-2023 18:37:31.438 +0000 INFO  loader [27902 HTTPDispatch] - Limiting REST HTTP server to 21845 sockets
06-07-2023 18:37:31.438 +0000 INFO  loader [27902 HTTPDispatch] - Limiting REST HTTP server to 161 threads
06-07-2023 18:37:31.438 +0000 WARN  X509Verify [27902 HTTPDispatch] - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>
06-07-2023 18:37:32.194 +0000 INFO  UiHttpListener [28468 WebuiStartup] - Server supporting SSL versions TLS1.2
06-07-2023 18:37:32.194 +0000 INFO  UiHttpListener [28468 WebuiStartup] - Using cipher suite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
06-07-2023 18:37:32.194 +0000 INFO  UiHttpListener [28468 WebuiStartup] - Using ECDH curves : prime256v1, secp384r1, secp521r1
06-07-2023 18:37:32.197 +0000 WARN  X509Verify [28468 WebuiStartup] - X509 certificate (O=SplunkUser,CN=ip-172-31-46-102.us-west-2.compute.internal) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates>
06-07-2023 18:37:32.197 +0000 INFO  UiHttpListener [28468 WebuiStartup] - Limiting UI HTTP server to 21845 sockets
06-07-2023 18:37:32.197 +0000 INFO  UiHttpListener [28468 WebuiStartup] - Limiting UI HTTP server to 161 threads
06-07-2023 18:37:32.251 +0000 INFO  DatabaseDirectoryManager [28321 IndexerService] - idx=_audit writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/audit/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='IndexerService periodic manifest update'
06-07-2023 18:37:32.252 +0000 INFO  DatabaseDirectoryManager [28321 IndexerService] - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/audit/db duration=0.001
06-07-2023 18:37:32.309 +0000 INFO  ProxyConfig [28468 WebuiStartup] - Failed to initialize http_proxy from server.conf for splunkd. Please make sure that the http_proxy property is set as http_proxy=http://host:port in case HTTP proxying needs to be enabled.
06-07-2023 18:37:32.310 +0000 INFO  ProxyConfig [28468 WebuiStartup] - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled.
06-07-2023 18:37:32.310 +0000 INFO  ProxyConfig [28468 WebuiStartup] - Failed to initialize the proxy_rules setting from server.conf for splunkd. Please provide a valid set of proxy_rules in case HTTP proxying needs to be enabled.
06-07-2023 18:37:32.310 +0000 INFO  ProxyConfig [28468 WebuiStartup] - Failed to initialize the no_proxy setting from server.conf for splunkd. Please provide a valid set of no_proxy rules in case HTTP proxying needs to be enabled.
06-07-2023 18:37:32.314 +0000 WARN  SSLOptions [28468 WebuiStartup] - <internal>.conf/[<internal>]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
06-07-2023 18:37:32.414 +0000 WARN  SSLOptions [28468 WebuiStartup] - <internal>.conf/[<internal>]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
06-07-2023 18:37:32.837 +0000 WARN  SSLOptions [28394 SchedulerThread] - server.conf/[search_state]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
06-07-2023 18:37:32.999 +0000 WARN  ProcessTracker [27894 MainThread] - (child_1__Fsck)  SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
06-07-2023 18:37:34.574 +0000 INFO  ExecProcessor [28362 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" splunk-dashboard-studio version is 1.7.3
06-07-2023 18:37:34.575 +0000 INFO  ExecProcessor [28362 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" Content of /opt/splunk/etc/apps/splunk-dashboard-studio/kvstore_icon_status.conf is {'default': {'uploadedVersion': '1.7.3'}}
06-07-2023 18:37:34.575 +0000 INFO  ExecProcessor [28362 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" Icons of splunk-dashboard-studio version 1.7.3 are already stored in kvstore collection. Skipping now and exiting.

[isoutamo]

Hi

i don’t think that those will explain, why splunkd doesn’t start. What else you have on logs after those?

r. Ismo

[PickleRick]

This just shows that _something_ went wrong. We don't know what. Check your log - /opt/splunk/var/log/splunk/splunkd.log