Getting Data In

Failed to set up Universal forwarder with docker compose

rotemya
Explorer

I want to setup a universal forwarder that receive logs from a syslog server (share a volume) and send logs to a receiver.
For some reason I get the error below on my forwarder container:

splunk-forwarder_1 | TASK
[splunk_universal_forwarder : Disable indexing on the current node] *
*****
splunk-forwarder_1 | fatal:> [localhost]: FAILED! =>> {"cache_control": "no-store, no-cache,> must-revalidate, max-age=0", "changed": false, "connection": "Close", "content": "\n\n \n \n In handler 'conf-outputs': Could not flush changes to disk: /nobody/system/outputs/indexAndForward/index:
ConfPathMapper: /opt/splunkforwarder/etc/system/local\n \n\n", "content_length": "279", "content_type": "text/xml; charset=UTF-8", "date": "Tue, 06 Aug 2019 08:23:31 GMT", "elapsed": 0, "expires": "Thu, 26 Oct 1978 00:00:00 GMT", "msg": "Status code was 500 and not [201, 409]: HTTP Error 500: Internal Server Error", "redirected":false, "server": "Splunkd", "status":500, "url":"https://127.0.0.1:8089/servicesNS/nobody/system/configs/conf-outputs","vary": "Cookie, Authorization","x_content_type_options": "nosniff", "x_frame_options": "SAMEORIGIN"}*

The outputs.conf on the forwarder:

[tcpout:splunkreceiver]
server=splunkreceiver:9997

When I remove this file - the error is gone, so I guess the problem is in this file.

My docker-compose.yml:

syslog-server:
  build: './collector'
  ports:
    - '8081:8081'
  volumes:
    - syslog-logs:/var/log/syslog-ng
  depends_on:
    - splunk-forwarder

splunk-forwarder:
  hostname: splunkuniversalforwarder
  image : splunk/universalforwarder
  ports:
    - '8082:8082'
  volumes:
    - ./forwarder/inputs.conf:/opt/splunkforwarder/etc/system/local/inputs.conf
    - ./forwarder/outputs.conf:/opt/splunkforwarder/etc/system/local/outputs.conf
    - syslog-logs:/opt/splunkforwarder/var/log
  env_file:
    - ./forwarder/forwarder.env
  depends_on:
    - splunk-receiver

splunk-receiver:
  hostname: splunkreceiver
  image : splunk/splunk:latest
  ports:
    - '8083:8083'
  env_file:
    - ./receiver/receiver.env
  volumes:
  - ./receiver/inputs.conf:/opt/splunk/etc/system/local/inputs.conf

Any Ideas?


More files:

The inputs.conf on the forwarder:

[monitor:///opt/splunkforwarder/var/log]
index=my-index
sourcetype=my-source-type
disabled = 0

The inputs.conf on the receiver:

[splunktcp://9997]
disabled = 0
0 Karma

mattymo
Splunk Employee
Splunk Employee

Hey @rotemya,

So there are a few suggestions I have here.

First off, all the non standard ports being opened in the compose do not map to the ports we use by default in the official docker image.

These ports can be set in the default.yml which is an important concept you need to review:

https://github.com/splunk/docker-splunk/blob/master/docs/ADVANCED.md

I recommend you pull the latest UF Image then run the default.yml generation command:

https://github.com/splunk/docker-splunk/blob/master/docs/ADVANCED.md#generation

This will spit out a valid default.yml where you can customize things like the splunk listener ports etc.

Secondly, I do not suggest copying files into the container the way you are doing. All config should be deployed as an “app”. which you can ensure is installed when the container spins up:

https://github.com/splunk/docker-splunk/blob/master/docs/advanced/APP_INSTALL.md

Here is an example I use to pull down the splunk image, then build one where I seed my default.yml and any apps I want installed:

https://github.com/matthewmodestino/container_workshop/tree/master/container_images/container_worksh...

There is also some sample docker compose examples in that repo.

Once you have successfully generated the default.yml and updated it accordingly, put your configs in a splunk app folder, tar it up and use my Dockerfile as an example to create your custom image, then update your compose and re-try.

If you need any clarification on any of that (after you read the documentation) let me know, and I can help you get deployed.

You can also catch me ( @mattymo ) on our community Slack channel here splk.it/slack

- MattyMo

rotemya
Explorer

Thanks @mmodestino_splunk For your solution.
It seems to me like a quiet non trivial approach which might be suitable for an experience splunk developer like yourself (:
But in my case, I only want to get the forwarder image up and running with a simple configuration like I did in the inputs.conf and outputs.conf and without splitting into separate apps and using TAR (I admit, this approach gave me quiet a headache - Is it a common approach?)

0 Karma

mattymo
Splunk Employee
Splunk Employee

you can doooo it!

its all documented quite extensively, and is the best practice of splunk app dev to keep your config modular and easy to troubleshoot

http://dev.splunk.com/view/get-started/SP-CAAAESC

If you want a quick fix, check out the SPLUNK_ADD variable and list your inputs there.

I do not advise this as a long term answer, but is ok to get started

- MattyMo
0 Karma

woodcock
Esteemed Legend

@mmodestino_splunk is the Duke of Docker at Splunk. He can help.

0 Karma

rotemya
Explorer

Hi, @mmodestino_splunk , Any idea?

0 Karma

mattymo
Splunk Employee
Splunk Employee

Sure, I can probably get you rocking.

- MattyMo

DavidHourani
Super Champion

Seems like a permission issue... Is the port 9997 already used ?

rotemya
Explorer

Port 9997 is exposed from the container.

docker-compose ps:
splunk_splunk-forwarder_1   /sbin/entrypoint.sh start- ...   Up      0.0.0.0:8082->8082/tcp, 8088/tcp, 8089/tcp, 9997/tcp     

But beside that , I don't think its in use - I can't debug it because the container crash after about 30 seconds.

0 Karma

DavidHourani
Super Champion

And im guessing if you try to setup the forwarder alone following this: https://github.com/splunk/docker-splunk-legacy/tree/master/universalforwarder It works, right ?

rotemya
Explorer

@David, First of all thank you. I'm not familiar with that image - I will try it now and Update.

0 Karma

rotemya
Explorer

@DavidHourani , it seems like the exact docker image - "splunk/universalforwarder".

0 Karma

DavidHourani
Super Champion

yeah it should be, try calling it alone, see if you have this problem. There's a compose example in the link.

0 Karma

rotemya
Explorer

This is quite embarrassing..
Copy and pasted the compose.yaml file , rebuilt the image and Eeecuted it along - Still failing:

              Name                             Command               State    Ports
-----------------------------------------------------------------------------------
splunk_splunkuniversalforwarder_1   /sbin/entrypoint.sh start- ...   Exit 1   
0 Karma

DavidHourani
Super Champion

This block is not working ??

volumes:
  opt-splunk-etc:
  opt-splunk-var:

services:
  splunkuniversalforwarder:

    hostname: splunkuniversalforwarder
    image: splunk/universalforwarder:7.0.3
    environment: SPLUNK_START_ARGS: --accept-license
    volumes:
      - opt-splunk-etc:/opt/splunk/etc
      - opt-splunk-var:/opt/splunk/var
    ports:
      - "8000:8000"
      - "9997:9997"
      - "8088:8088"
      - "1514:1514"
0 Karma

rotemya
Explorer

With the addition of the first line from the example:

version: '3'

No, it is not working, I'm sorry.

0 Karma

mattymo
Splunk Employee
Splunk Employee

supported containers are 7.2+. Anything earlier I recommend you dont use!

- MattyMo
0 Karma

rotemya
Explorer

you mean an image of splunk/universalforwarder:7.2 and above?

0 Karma

mattymo
Splunk Employee
Splunk Employee

correct. Tags 7.2+

- MattyMo
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...