splunkd was not running. Waiting for web server

SOCat — Wed, 21 Sep 2022 18:23:04 GMT

Hello,

I've tried figuring this out on my own but I couldn't find any related threads which would fixed my problems.

I'm trying to install Splunk enterprise on my Ubuntu 20.04.5 LTS Server as root.

I've tried both .deb and .tar versions by following the docs . I've also tried following the new installation manual video.

After starting splunk with ./splunk start and accepting the license I was prompted to rename the default account, i continued with enter to use the default admin name. Then I changed the default password and waited for the RSA key gen and preliminary checks.

After that I am prompted with:

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done

Waiting for web server at http://127.0.0.1:8000 to be available........splunkd 4932 was not running.

Stopping splunk helpers...

Done.

Stopped helpers.

Removing stale pid file... done.

WARNING: web interface does not seem to be available!

I've also checked the logs but could't figure out the problem on my own:

Last entries of cat /opt/splunk/var/log/splunk/splunkd.log

09-21-2022 19:55:57.924 +0200 INFO  PipelineComponent [4932 MainThread] - Pipeline vix disabled in default-mode.conf file
09-21-2022 19:55:57.932 +0200 WARN  IntrospectionGenerator:resource_usage [5097 ExecProcessor] -   SSLOptions - server.conf/[sslConfig]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
09-21-2022 19:55:57.942 +0200 WARN  Thread [4932 MainThread] - MainThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 55 threads active. Trying to create QueueServiceThread
09-21-2022 19:55:57.944 +0200 WARN  IntrospectionGenerator:resource_usage [5097 ExecProcessor] -   SSLCommon - PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
09-21-2022 19:55:57.945 +0200 ERROR ExecProcessor [5097 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk-dashboard-studio/bin/save_image_and_icon_on_install.py" /bin/sh: 1: Cannot fork
09-21-2022 19:55:57.945 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/selfupdate_modular_input.py": Resource temporarily unavailable
09-21-2022 19:55:57.948 +0200 WARN  IntrospectionGenerator:resource_usage [5097 ExecProcessor] -   SSLOptions - server.conf/[kvstore]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
09-21-2022 19:55:57.949 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/supervisor_modular_input.py": Resource temporarily unavailable
09-21-2022 19:55:57.952 +0200 WARN  IntrospectionGenerator:resource_usage [5097 ExecProcessor] -   Thread - MainThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 2 threads active. Trying to create KVStoreServerStatusInstrumentThread
09-21-2022 19:55:57.953 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py": Resource temporarily unavailable
09-21-2022 19:55:57.954 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_instrumentation/bin/on_splunk_start.py": Resource temporarily unavailable
09-21-2022 19:55:57.954 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py": Resource temporarily unavailable
09-21-2022 19:55:57.954 +0200 INFO  IntrospectionGenerator:resource_usage [5097 ExecProcessor] -  terminate called after throwing an instance of '15ThreadException'
09-21-2022 19:55:57.955 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_monitoring_console/bin/mc_auto_config.py": Resource temporarily unavailable
09-21-2022 19:55:57.956 +0200 ERROR ExecProcessor [5097 ExecProcessor] - Couldn't start command "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_secure_gateway/bin/ssg_enable_modular_input.py": Resource temporarily unavailable
09-21-2022 19:55:57.956 +0200 INFO  IntrospectionGenerator:resource_usage [5097 ExecProcessor] -    what():  MainThread: about to throw a ThreadException: pthread_create: Resource temporarily unavailable; 2 threads active. Trying to create KVStoreServerStatusInstrumentThread

ulimit -n -u

open files                      (-n) 1024
max user processes              (-u) 62987

Does anyone know what I am doing wrong?

Please help me I have no spluck>

: (

Re: splunkd was not running. Waiting for web server

isoutamo — Wed, 21 Sep 2022 20:31:55 GMT

based on log, you are running out of resources. Maybe this explain it

Information on expected default shell and caveats for Debian shells

On later versions of Debian Linux (for example, Debian Squeeze), the default non-interactive shell is the dash shell. Splunk Enterprise expects to run commands using the bash shell, and bash to be available from /bin/sh. Using the dash shell can result in zombie processes - processes that have completed execution, yet remain in the process table and cannot be killed or removed. If you run Debian Linux, consider changing your default shell to be bash.

Also open files should be much higher like 64k. https://community.splunk.com/t5/Splunk-Search/Changing-the-Ulimits-for-openfiles/m-p/318521
r. Ismo

topic splunkd was not running. Waiting for web server in Installation

splunkd was not running. Waiting for web server

Re: splunkd was not running. Waiting for web server

Information on expected default shell and caveats for Debian shells