Installation

How to resolve error when starting UF for the first time?

korix
New Member

Hello to everyone,

My environment :

A part of my infrastructure is deployed as Docker containers that are built and configured by myself. Bassicaly I'm pulling an ubuntu:latest image on which i'm installing a splunk forwarder which will transfer logs to a central splunk enterprise.

On start time I'm using supervisorD which is a process control system to start the UF and other processes.

 

The following steps are done on every build / deployment :

- Pulling the latest image of Ubuntu

- Installing / Configuring splunk forwarder (creating user / downloading . deb / installing)

- Installing other packages

- Starting Docker container with a simple bash script that do things at runtime.

- Starting the service via supervisorD as root which then start the UF as splunk user.

 

Configurations

My Docker File configuration.

 

 

 

FROM ubuntu:latest
ENV TZ=Europe/Paris
ARG DEBIAN_FRONTEND=noninteractive
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN adduser --home /home/www-python --disabled-password --gecos "" www-python \
   && groupadd -r splunk \
   && useradd -r -m -g splunk splunk \
   && apt update \
   && apt install -y python3 python3-pip wget curl supervisor
   
RUN wget -O splunkforwarder-8.2.8-da25d08d5d3e-linux-2.6-amd64.deb "https://download.splunk.com/products/universalforwarder/releases/8.2.8/linux/splunkforwarder-8.2.8-da25d08d5d3e-linux-2.6-amd64.deb" \
&& dpkg -i splunkforwarder-*.deb \
&& rm -f splunkforwarder-*
   
COPY [ "src/splunkforwarder/inputs.conf", "src/splunkforwarder/outputs.conf", "src/splunkforwarder/server.conf", "/opt/splunkforwarder/etc/system/local/" ]
   
USER root
WORKDIR /root/
COPY [ "src/supervisor/service.conf", "/root/"]
COPY ./src/start.sh /root/
RUN chmod +x /root/start.sh

 

 

 

 

My start.sh script.

 

 

 

#!/bin/bash

#Doing runtime stuff

supervisord -c /root/service.conf

 

 

 

 

My supervisor configuration.

 

 

 

[supervisord]
nodaemon=true
user=root

[program:splunkforwarder]
command=/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt
user=splunk

[program:python-script]
command=some command to start a service

 

 

 

 

Problem encountered

I've traced back the problem to the version 9.0.0. So all the steps and configuration in this post is working on all versions under 9.0.0.

Version 9.0.1 is also having the same behavior.

When the container starts, my supervisord indicates that everything started smoothly.

 

 

 

2022-09-12 09:15:17,813 INFO Set uid to user 0 succeeded
2022-09-12 09:15:17,844 INFO supervisord started with pid 10
2022-09-12 09:15:18,856 INFO spawned: 'python-script' with pid 11
2022-09-12 09:15:18,858 INFO spawned: 'splunkforwarder' with pid 12
2022-09-12 09:15:19,863 INFO success: python-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-09-12 09:15:19,863 INFO success: splunkforwarder entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)

 

 

 

 

But the catch starts here, the splunk daemon seems to be stuck. As you can see my ps aux indicates that the service is taking all the CPU.

 

 

 

root@demo:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2904  1012 ?        Ss   09:15   0:00 /bin/sh -c /root/start.sh
root         7  0.0  0.0   4508  3516 ?        S    09:15   0:00 /bin/bash /root/start.sh
root        10  0.0  0.1  33220 27204 ?        S    09:15   0:02 /usr/bin/python3 /usr/bin/supervisord -c /root/service.conf
splunk      12 56.0  0.0   4516  2900 ?        R    09:15 145:55 /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt

 

 

 

 

When you go look for more logs, nothings is created, like the service never did start.

 

 

 

root@demo:~# ls -alh /opt/splunkforwarder/var/log/splunk/
total 4.0K
drwx------ 2 splunk splunk 31 Sep 12 09:15 .
drwx--x--- 5 splunk splunk 57 Sep 12 09:15 ..
-rw------- 1 splunk splunk 70 Sep 12 09:15 first_install.log

 

 

 

 

So if I try to start manualy the service, it asks me to accept the licence agreements & indicates that a previous installion has been found and needs to migrate the instance.

The only logs of errors I have are these lines :

 

 

 

Creating unit file...
Error calling execve(): No such file or directory
Error launching  command: No such file or directory

 

 

 

 

Like I was saying earlier, the only thing that changed on this case is the version of splunk. I can start the service by using version 8.2.X but not the latest one.

Does anyone have any inputs on this matter ? I didn't had any insight by looking at the threads on this site (or elsewhere)

Labels (2)
0 Karma

GayathiriKCisco
Loves-to-Learn

Where you able to fix this?

0 Karma

tkoster8
New Member

So I ran into the same problem. 

 

For me it was that I was running kernel 2.6.X and 9.0 stopped supporting that. 

 

Check your kernel version and see if thats the issue (uname -msr ) 

supported version for 8.2: https://docs.splunk.com/Documentation/Splunk/8.2.5/Installation/Systemrequirements

vs 9.0.2

https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements?_ga=2.37568416.1....

 

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...