Why are we unable to run splunk forwarder image as non root user?

[tsfadz]

Hi,

I want to run splunk-universalforwarder with non-root user. I created my own docker image and tried to run it.
But when I run the docker image and it return error as "sh: 1: cannot create /opt/container_artifact/splunk-container.state: Permission denied".

Details as below:
- Dockerfile

FROM splunk/universalforwarder:latest
USER 9999

• Running command

  docker build ./ -t uf_9999
  Sending build context to Docker daemon  1.723MB
  Step 1/2 : FROM splunk/universalforwarder:latest
  latest: Pulling from splunk/universalforwarder
  27833a3ba0a5: Already exists
  669cca39e74e: Pull complete
  4fb681d8edc0: Pull complete
  3056b53853fb: Pull complete
  660149c27f87: Pull complete
  e471a3f1dc9a: Pull complete
  5f8d48462400: Pull complete
  166bfc206ee8: Pull complete
  5ebb6bc9ea16: Pull complete
  Digest: sha256:37d48e3733cf3d987d37401268f341c48e1dda6f7fc68281b8aeaa40f84b0135
  Status: Downloaded newer image for splunk/universalforwarder:latest
   ---> 1adf45780dc6
  Step 2/2 : USER 9999
   ---> Running in 97fbfea0e6ae
  Removing intermediate container 97fbfea0e6ae
   ---> cb653fda417a
  Successfully built cb653fda417a
  Successfully tagged uf_9999:latest
  SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
  
  λ docker images
  REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
  uf_9999                     latest              cb653fda417a        27 seconds ago      219MB
  

  λ docker run -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=splunk" cb653fda417a
  WARNING: No password ENV var. Stack may fail to provision if splunk.password is not set in ENV or a default.yml
  sh: 1: cannot create /opt/container_artifact/splunk-container.state: Permission denied

I did tried change USER to 999 but it resulted the same.

Does anyone know how could i resolve this error?

Thanks

[codebuilder · Accepted solution]

In your Docker image /opt is owned by root, so the splunk user cannot write there.
Create /opt/splunk if it does not exist and recursively change ownerships
chown -RP splunk:splunk /opt/splunk

You can also run your image as root so that it can write to /opt as such:
docker run -d -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=root" -p image_name:image_version

Or you can rebuild your image so that /opt/splunk is created and owned by the splunk user.
To do so, ensure that your DockerFile contains the following lines in the RUN command, and rebuild it (the mkdir and chown's are what you really need to focus on):

RUN mkdir -p ${SPLUNK_HOME} \
    && wget -qO /tmp/${SPLUNK_FILENAME} http://<remote_host>/${SPLUNK_FILENAME} \
    && tar xzf /tmp/${SPLUNK_FILENAME} --strip 1 -C ${SPLUNK_HOME} \
    && rm /tmp/${SPLUNK_FILENAME} \
    && rm /tmp/${SPLUNK_FILENAME}.md5 \
    && mkdir -p /var/opt/splunk \
    && cp -R ${SPLUNK_HOME}/etc ${SPLUNK_BACKUP_DEFAULT_ETC} \
    && rm -fR ${SPLUNK_HOME}/etc \
    && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_HOME} \
    && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_BACKUP_DEFAULT_ETC} \

I've used both methods with success. Rebuilding with the above lines allowed me to deploy to Kubernetes without issue.
In my deployment yaml I have set the following under spec/containers:

    volumeMounts:
    - mountPath: /opt/splunk/var/run

This should get you on the right track.

[tsfadz]

"spec:
      serviceAccount: tenant-pod-nonroot
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      - name: splunk
        image: splunk/universalforwarder:latest
        env:
        - name: TZ
          value: ""Asia/Tokyo""
        - name: SPLUNK_USER
          value: ""root""
        - name: SPLUNK_START_ARGS
          value: ""--accept-license --seed-passwd changeme"""

This is my deployment manifest that I'm using (based on your advice to use root). But it resulted as below:
sh: 1: cannot create /opt/container_artifact/splunk-container.state: Permission denied

[codebuilder]

I also just noticed that your serviceAccount is labeled as "tenant-pod-nonroot", but your SPLUNK_USER is set to root. Without knowing your environment, that sounds like a conflict as well.

[tsfadz]

I thought you say OR, that's why i put normal image and use root user.
Will try again with build image then.
Is mount path really need to be /opt/splunk?
I configured it to different path that /opt/splunk (i used different path to collect log files only)

[tsfadz]

Thank you @codebuilder for helping me out. Managed to build it successfully in my local.
Couldn't make it into kubernetes yet, due to some security policy to download certain stuff.
But I guess it should work the same as my local.

Thank you again ^__^

[codebuilder]

Glad to help and good to hear you got it working!

[codebuilder]

You'll need to rebuild the image using the settings I mentioned in the DockerFile.

The image that you are pulling down in your deployment yaml is using what is pre-built by Splunk.

image: splunk/universalforwarder:latest

Also I don't see your mountPath listed, what is that set to?

[mattymo]

any reason not to use the splunk supported image instead?

https://hub.docker.com/r/splunk/universalforwarder

It runs splunk as splunk user, but has an ansible user that has some sudo.

[tsfadz]

I did try to use splunk supported image previously and it resulted as below:
cp: cannot create regular file ‘/opt/splunk/etc/copyright.txt’: Permission denied
cp: cannot create regular file ‘/opt/splunk/etc/datetime.xml’: Permission denied
cp: cannot create regular file ‘/opt/splunk/etc/regid.2001-12.com.splunk-UniversalForwarder.swidtag’: Permission denied
and so on (total around 13 permission denied error)

That's the reason i tried to create new one instead supported one.
By the way, the main reason is because I need to deploy splunk forwarder into kubernetes which the current pod required container to run as non root user.