Getting Data In

Why are we unable to run splunk forwarder image as non root user?

New Member

Hi,

I want to run splunk-universalforwarder with non-root user. I created my own docker image and tried to run it.
But when I run the docker image and it return error as "sh: 1: cannot create /opt/container_artifact/splunk-container.state: Permission denied".

Details as below:
- Dockerfile

FROM splunk/universalforwarder:latest
USER 9999
  • Running command

    docker build ./ -t uf_9999
    Sending build context to Docker daemon  1.723MB
    Step 1/2 : FROM splunk/universalforwarder:latest
    latest: Pulling from splunk/universalforwarder
    27833a3ba0a5: Already exists
    669cca39e74e: Pull complete
    4fb681d8edc0: Pull complete
    3056b53853fb: Pull complete
    660149c27f87: Pull complete
    e471a3f1dc9a: Pull complete
    5f8d48462400: Pull complete
    166bfc206ee8: Pull complete
    5ebb6bc9ea16: Pull complete
    Digest: sha256:37d48e3733cf3d987d37401268f341c48e1dda6f7fc68281b8aeaa40f84b0135
    Status: Downloaded newer image for splunk/universalforwarder:latest
     ---> 1adf45780dc6
    Step 2/2 : USER 9999
     ---> Running in 97fbfea0e6ae
    Removing intermediate container 97fbfea0e6ae
     ---> cb653fda417a
    Successfully built cb653fda417a
    Successfully tagged uf_9999:latest
    SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
    
    λ docker images
    REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE
    uf_9999                     latest              cb653fda417a        27 seconds ago      219MB
    

    λ docker run -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=splunk" cb653fda417a
    WARNING: No password ENV var. Stack may fail to provision if splunk.password is not set in ENV or a default.yml
    sh: 1: cannot create /opt/container_artifact/splunk-container.state: Permission denied

I did tried change USER to 999 but it resulted the same.

Does anyone know how could i resolve this error?

Thanks

0 Karma
1 Solution

Motivator

In your Docker image /opt is owned by root, so the splunk user cannot write there.
Create /opt/splunk if it does not exist and recursively change ownerships
chown -RP splunk:splunk /opt/splunk

You can also run your image as root so that it can write to /opt as such:
docker run -d -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=root" -p image_name:image_version

Or you can rebuild your image so that /opt/splunk is created and owned by the splunk user.
To do so, ensure that your DockerFile contains the following lines in the RUN command, and rebuild it (the mkdir and chown's are what you really need to focus on):

RUN mkdir -p ${SPLUNK_HOME} \
    && wget -qO /tmp/${SPLUNK_FILENAME} http://<remote_host>/${SPLUNK_FILENAME} \
    && tar xzf /tmp/${SPLUNK_FILENAME} --strip 1 -C ${SPLUNK_HOME} \
    && rm /tmp/${SPLUNK_FILENAME} \
    && rm /tmp/${SPLUNK_FILENAME}.md5 \
    && mkdir -p /var/opt/splunk \
    && cp -R ${SPLUNK_HOME}/etc ${SPLUNK_BACKUP_DEFAULT_ETC} \
    && rm -fR ${SPLUNK_HOME}/etc \
    && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_HOME} \
    && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_BACKUP_DEFAULT_ETC} \

I've used both methods with success. Rebuilding with the above lines allowed me to deploy to Kubernetes without issue.
In my deployment yaml I have set the following under spec/containers:

    volumeMounts:
    - mountPath: /opt/splunk/var/run

This should get you on the right track.

View solution in original post

0 Karma

Motivator

In your Docker image /opt is owned by root, so the splunk user cannot write there.
Create /opt/splunk if it does not exist and recursively change ownerships
chown -RP splunk:splunk /opt/splunk

You can also run your image as root so that it can write to /opt as such:
docker run -d -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_USER=root" -p image_name:image_version

Or you can rebuild your image so that /opt/splunk is created and owned by the splunk user.
To do so, ensure that your DockerFile contains the following lines in the RUN command, and rebuild it (the mkdir and chown's are what you really need to focus on):

RUN mkdir -p ${SPLUNK_HOME} \
    && wget -qO /tmp/${SPLUNK_FILENAME} http://<remote_host>/${SPLUNK_FILENAME} \
    && tar xzf /tmp/${SPLUNK_FILENAME} --strip 1 -C ${SPLUNK_HOME} \
    && rm /tmp/${SPLUNK_FILENAME} \
    && rm /tmp/${SPLUNK_FILENAME}.md5 \
    && mkdir -p /var/opt/splunk \
    && cp -R ${SPLUNK_HOME}/etc ${SPLUNK_BACKUP_DEFAULT_ETC} \
    && rm -fR ${SPLUNK_HOME}/etc \
    && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_HOME} \
    && chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_BACKUP_DEFAULT_ETC} \

I've used both methods with success. Rebuilding with the above lines allowed me to deploy to Kubernetes without issue.
In my deployment yaml I have set the following under spec/containers:

    volumeMounts:
    - mountPath: /opt/splunk/var/run

This should get you on the right track.

View solution in original post

0 Karma

New Member
"spec:
      serviceAccount: tenant-pod-nonroot
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      - name: splunk
        image: splunk/universalforwarder:latest
        env:
        - name: TZ
          value: ""Asia/Tokyo""
        - name: SPLUNK_USER
          value: ""root""
        - name: SPLUNK_START_ARGS
          value: ""--accept-license --seed-passwd changeme"""

This is my deployment manifest that I'm using (based on your advice to use root). But it resulted as below:
sh: 1: cannot create /opt/container_artifact/splunk-container.state: Permission denied

0 Karma

Motivator

I also just noticed that your serviceAccount is labeled as "tenant-pod-nonroot", but your SPLUNK_USER is set to root. Without knowing your environment, that sounds like a conflict as well.

0 Karma

New Member

I thought you say OR, that's why i put normal image and use root user.
Will try again with build image then.
Is mount path really need to be /opt/splunk?
I configured it to different path that /opt/splunk (i used different path to collect log files only)

0 Karma

New Member

Thank you @codebuilder for helping me out. Managed to build it successfully in my local.
Couldn't make it into kubernetes yet, due to some security policy to download certain stuff.
But I guess it should work the same as my local.

Thank you again ^__^

0 Karma

Motivator

Glad to help and good to hear you got it working!

0 Karma

Motivator

You'll need to rebuild the image using the settings I mentioned in the DockerFile.

The image that you are pulling down in your deployment yaml is using what is pre-built by Splunk.

image: splunk/universalforwarder:latest

Also I don't see your mountPath listed, what is that set to?

0 Karma

Splunk Employee
Splunk Employee

any reason not to use the splunk supported image instead?

https://hub.docker.com/r/splunk/universalforwarder

It runs splunk as splunk user, but has an ansible user that has some sudo.

0 Karma

New Member

I did try to use splunk supported image previously and it resulted as below:
cp: cannot create regular file ‘/opt/splunk/etc/copyright.txt’: Permission denied
cp: cannot create regular file ‘/opt/splunk/etc/datetime.xml’: Permission denied
cp: cannot create regular file ‘/opt/splunk/etc/regid.2001-12.com.splunk-UniversalForwarder.swidtag’: Permission denied
and so on (total around 13 permission denied error)

That's the reason i tried to create new one instead supported one.
By the way, the main reason is because I need to deploy splunk forwarder into kubernetes which the current pod required container to run as non root user.

0 Karma