topic Re: Failed to set up Universal forwarder with docker compose in Getting Data In

Failed to set up Universal forwarder with docker compose

rotemya — Wed, 30 Sep 2020 01:36:56 GMT

I want to setup a universal forwarder that receive logs from a syslog server (share a volume) and send logs to a receiver.
For some reason I get the error below on my forwarder container:

splunk-forwarder_1 | TASK
[splunk_universal_forwarder : Disable indexing on the current node] ******
splunk-forwarder_1 | fatal:> [localhost]: FAILED! =>> {"cache_control": "no-store, no-cache,> must-revalidate, max-age=0", "changed": false, "connection": "Close", "content": "\n\n \n \n In handler 'conf-outputs': Could not flush changes to disk: /nobody/system/outputs/indexAndForward/index:
ConfPathMapper: /opt/splunkforwarder/etc/system/local\n \n\n", "content_length": "279", "content_type": "text/xml; charset=UTF-8", "date": "Tue, 06 Aug 2019 08:23:31 GMT", "elapsed": 0, "expires": "Thu, 26 Oct 1978 00:00:00 GMT", "msg": "Status code was 500 and not [201, 409]: HTTP Error 500: Internal Server Error", "redirected":false, "server": "Splunkd", "status":500, "url":"https://127.0.0.1:8089/servicesNS/nobody/system/configs/conf-outputs","vary": "Cookie, Authorization","x_content_type_options": "nosniff", "x_frame_options": "SAMEORIGIN"}*

The outputs.conf on the forwarder:

[tcpout:splunkreceiver]
server=splunkreceiver:9997

When I remove this file - the error is gone, so I guess the problem is in this file.

My docker-compose.yml:

syslog-server:
  build: './collector'
  ports:
    - '8081:8081'
  volumes:
    - syslog-logs:/var/log/syslog-ng
  depends_on:
    - splunk-forwarder

splunk-forwarder:
  hostname: splunkuniversalforwarder
  image : splunk/universalforwarder
  ports:
    - '8082:8082'
  volumes:
    - ./forwarder/inputs.conf:/opt/splunkforwarder/etc/system/local/inputs.conf
    - ./forwarder/outputs.conf:/opt/splunkforwarder/etc/system/local/outputs.conf
    - syslog-logs:/opt/splunkforwarder/var/log
  env_file:
    - ./forwarder/forwarder.env
  depends_on:
    - splunk-receiver

splunk-receiver:
  hostname: splunkreceiver
  image : splunk/splunk:latest
  ports:
    - '8083:8083'
  env_file:
    - ./receiver/receiver.env
  volumes:
  - ./receiver/inputs.conf:/opt/splunk/etc/system/local/inputs.conf

Any Ideas?

More files:

The inputs.conf on the forwarder:

[monitor:///opt/splunkforwarder/var/log]
index=my-index
sourcetype=my-source-type
disabled = 0

The inputs.conf on the receiver:

[splunktcp://9997]
disabled = 0

Re: Failed to set up Universal forwarder with docker compose

DavidHourani — Tue, 06 Aug 2019 08:52:17 GMT

Seems like a permission issue... Is the port 9997 already used ?

Re: Failed to set up Universal forwarder with docker compose

rotemya — Tue, 06 Aug 2019 09:54:36 GMT

Port 9997 is exposed from the container.

docker-compose ps:
splunk_splunk-forwarder_1   /sbin/entrypoint.sh start- ...   Up      0.0.0.0:8082->8082/tcp, 8088/tcp, 8089/tcp, 9997/tcp

But beside that , I don't think its in use - I can't debug it because the container crash after about 30 seconds.

Re: Failed to set up Universal forwarder with docker compose

DavidHourani — Tue, 06 Aug 2019 10:12:57 GMT

And im guessing if you try to setup the forwarder alone following this: https://github.com/splunk/docker-splunk-legacy/tree/master/universalforwarder It works, right ?

Re: Failed to set up Universal forwarder with docker compose

rotemya — Tue, 06 Aug 2019 12:24:55 GMT

@David, First of all thank you. I'm not familiar with that image - I will try it now and Update.

Re: Failed to set up Universal forwarder with docker compose

rotemya — Tue, 06 Aug 2019 13:02:19 GMT

@DavidHourani , it seems like the exact docker image - "splunk/universalforwarder".

Re: Failed to set up Universal forwarder with docker compose

DavidHourani — Tue, 06 Aug 2019 13:28:02 GMT

yeah it should be, try calling it alone, see if you have this problem. There's a compose example in the link.

Re: Failed to set up Universal forwarder with docker compose

rotemya — Tue, 06 Aug 2019 14:20:41 GMT

This is quite embarrassing..
Copy and pasted the compose.yaml file , rebuilt the image and Eeecuted it along - Still failing:

              Name                             Command               State    Ports
-----------------------------------------------------------------------------------
splunk_splunkuniversalforwarder_1   /sbin/entrypoint.sh start- ...   Exit 1

Re: Failed to set up Universal forwarder with docker compose

DavidHourani — Tue, 06 Aug 2019 14:40:38 GMT

This block is not working ??

volumes:
  opt-splunk-etc:
  opt-splunk-var:

services:
  splunkuniversalforwarder:

    hostname: splunkuniversalforwarder
    image: splunk/universalforwarder:7.0.3
    environment: SPLUNK_START_ARGS: --accept-license
    volumes:
      - opt-splunk-etc:/opt/splunk/etc
      - opt-splunk-var:/opt/splunk/var
    ports:
      - "8000:8000"
      - "9997:9997"
      - "8088:8088"
      - "1514:1514"

Re: Failed to set up Universal forwarder with docker compose

rotemya — Tue, 06 Aug 2019 15:01:02 GMT

With the addition of the first line from the example:

version: '3'

No, it is not working, I'm sorry.

Re: Failed to set up Universal forwarder with docker compose

woodcock — Tue, 06 Aug 2019 15:22:23 GMT

@mmodestino_splunk is the Duke of Docker at Splunk. He can help.

Re: Failed to set up Universal forwarder with docker compose

rotemya — Tue, 06 Aug 2019 17:12:28 GMT

Hi, @mmodestino_splunk , Any idea?

Re: Failed to set up Universal forwarder with docker compose

mattymo — Thu, 08 Aug 2019 16:19:41 GMT

Sure, I can probably get you rocking.

Re: Failed to set up Universal forwarder with docker compose

mattymo — Thu, 08 Aug 2019 16:34:42 GMT

Hey @rotemya,

So there are a few suggestions I have here.

First off, all the non standard ports being opened in the compose do not map to the ports we use by default in the official docker image.

These ports can be set in the default.yml which is an important concept you need to review:

https://github.com/splunk/docker-splunk/blob/master/docs/ADVANCED.md

I recommend you pull the latest UF Image then run the default.yml generation command:

https://github.com/splunk/docker-splunk/blob/master/docs/ADVANCED.md#generation

This will spit out a valid default.yml where you can customize things like the splunk listener ports etc.

Secondly, I do not suggest copying files into the container the way you are doing. All config should be deployed as an “app”. which you can ensure is installed when the container spins up:

https://github.com/splunk/docker-splunk/blob/master/docs/advanced/APP_INSTALL.md

Here is an example I use to pull down the splunk image, then build one where I seed my default.yml and any apps I want installed:

https://github.com/matthewmodestino/container_workshop/tree/master/container_images/container_workshop

There is also some sample docker compose examples in that repo.

Once you have successfully generated the default.yml and updated it accordingly, put your configs in a splunk app folder, tar it up and use my Dockerfile as an example to create your custom image, then update your compose and re-try.

If you need any clarification on any of that (after you read the documentation) let me know, and I can help you get deployed.

You can also catch me ( @mattymo ) on our community Slack channel here splk.it/slack

Re: Failed to set up Universal forwarder with docker compose

mattymo — Thu, 08 Aug 2019 16:44:32 GMT

supported containers are 7.2+. Anything earlier I recommend you dont use!

Re: Failed to set up Universal forwarder with docker compose

rotemya — Sat, 10 Aug 2019 19:07:32 GMT

you mean an image of splunk/universalforwarder:7.2 and above?

Re: Failed to set up Universal forwarder with docker compose

mattymo — Sat, 10 Aug 2019 22:18:42 GMT

correct. Tags 7.2+

Re: Failed to set up Universal forwarder with docker compose

rotemya — Mon, 12 Aug 2019 08:37:08 GMT

Thanks @mmodestino_splunk For your solution.
It seems to me like a quiet non trivial approach which might be suitable for an experience splunk developer like yourself (:
But in my case, I only want to get the forwarder image up and running with a simple configuration like I did in the inputs.conf and outputs.conf and without splitting into separate apps and using TAR (I admit, this approach gave me quiet a headache - Is it a common approach?)

Re: Failed to set up Universal forwarder with docker compose

mattymo — Tue, 13 Aug 2019 20:33:04 GMT

you can doooo it!

its all documented quite extensively, and is the best practice of splunk app dev to keep your config modular and easy to troubleshoot

http://dev.splunk.com/view/get-started/SP-CAAAESC

If you want a quick fix, check out the SPLUNK_ADD variable and list your inputs there.

I do not advise this as a long term answer, but is ok to get started