Getting Data In

Failed to set up Universal forwarder with docker compose

Explorer

I want to setup a universal forwarder that receive logs from a syslog server (share a volume) and send logs to a receiver.
For some reason I get the error below on my forwarder container:

splunk-forwarder1 | TASK
[splunk
universal_forwarder : Disable indexing on the current node] *
*****
splunk-forwarder1 | fatal:> [localhost]: FAILED! =>> {"cachecontrol": "no-store, no-cache,> must-revalidate, max-age=0", "changed": false, "connection": "Close", "content": "\n\n \n \n In handler 'conf-outputs': Could not flush changes to disk: /nobody/system/outputs/indexAndForward/index:
ConfPathMapper: /opt/splunkforwarder/etc/system/local\n \n\n", "contentlength": "279", "contenttype": "text/xml; charset=UTF-8", "date": "Tue, 06 Aug 2019 08:23:31 GMT", "elapsed": 0, "expires": "Thu, 26 Oct 1978 00:00:00 GMT", "msg": "Status code was 500 and not [201, 409]: HTTP Error 500: Internal Server Error", "redirected":false, "server": "Splunkd", "status":500, "url":"https://127.0.0.1:8089/servicesNS/nobody/system/configs/conf-outputs","vary": "Cookie, Authorization","xcontenttypeoptions": "nosniff", "xframe_options": "SAMEORIGIN"}*

The outputs.conf on the forwarder:

[tcpout:splunkreceiver]
server=splunkreceiver:9997

When I remove this file - the error is gone, so I guess the problem is in this file.

My docker-compose.yml:

syslog-server:
  build: './collector'
  ports:
    - '8081:8081'
  volumes:
    - syslog-logs:/var/log/syslog-ng
  depends_on:
    - splunk-forwarder

splunk-forwarder:
  hostname: splunkuniversalforwarder
  image : splunk/universalforwarder
  ports:
    - '8082:8082'
  volumes:
    - ./forwarder/inputs.conf:/opt/splunkforwarder/etc/system/local/inputs.conf
    - ./forwarder/outputs.conf:/opt/splunkforwarder/etc/system/local/outputs.conf
    - syslog-logs:/opt/splunkforwarder/var/log
  env_file:
    - ./forwarder/forwarder.env
  depends_on:
    - splunk-receiver

splunk-receiver:
  hostname: splunkreceiver
  image : splunk/splunk:latest
  ports:
    - '8083:8083'
  env_file:
    - ./receiver/receiver.env
  volumes:
  - ./receiver/inputs.conf:/opt/splunk/etc/system/local/inputs.conf

Any Ideas?


More files:

The inputs.conf on the forwarder:

[monitor:///opt/splunkforwarder/var/log]
index=my-index
sourcetype=my-source-type
disabled = 0

The inputs.conf on the receiver:

[splunktcp://9997]
disabled = 0
0 Karma

Splunk Employee
Splunk Employee

Hey @rotemya,

So there are a few suggestions I have here.

First off, all the non standard ports being opened in the compose do not map to the ports we use by default in the official docker image.

These ports can be set in the default.yml which is an important concept you need to review:

https://github.com/splunk/docker-splunk/blob/master/docs/ADVANCED.md

I recommend you pull the latest UF Image then run the default.yml generation command:

https://github.com/splunk/docker-splunk/blob/master/docs/ADVANCED.md#generation

This will spit out a valid default.yml where you can customize things like the splunk listener ports etc.

Secondly, I do not suggest copying files into the container the way you are doing. All config should be deployed as an “app”. which you can ensure is installed when the container spins up:

https://github.com/splunk/docker-splunk/blob/master/docs/advanced/APP_INSTALL.md

Here is an example I use to pull down the splunk image, then build one where I seed my default.yml and any apps I want installed:

https://github.com/matthewmodestino/container_workshop/tree/master/container_images/container_worksh...

There is also some sample docker compose examples in that repo.

Once you have successfully generated the default.yml and updated it accordingly, put your configs in a splunk app folder, tar it up and use my Dockerfile as an example to create your custom image, then update your compose and re-try.

If you need any clarification on any of that (after you read the documentation) let me know, and I can help you get deployed.

You can also catch me ( @mattymo ) on our community Slack channel here splk.it/slack

Explorer

Thanks @mmodestino_splunk For your solution.
It seems to me like a quiet non trivial approach which might be suitable for an experience splunk developer like yourself (:
But in my case, I only want to get the forwarder image up and running with a simple configuration like I did in the inputs.conf and outputs.conf and without splitting into separate apps and using TAR (I admit, this approach gave me quiet a headache - Is it a common approach?)

0 Karma

Splunk Employee
Splunk Employee

you can doooo it!

its all documented quite extensively, and is the best practice of splunk app dev to keep your config modular and easy to troubleshoot

http://dev.splunk.com/view/get-started/SP-CAAAESC

If you want a quick fix, check out the SPLUNK_ADD variable and list your inputs there.

I do not advise this as a long term answer, but is ok to get started

0 Karma

Esteemed Legend

@mmodestino_splunk is the Duke of Docker at Splunk. He can help.

0 Karma

Explorer

Hi, @mmodestino_splunk , Any idea?

0 Karma

Splunk Employee
Splunk Employee

Sure, I can probably get you rocking.

SplunkTrust
SplunkTrust

Seems like a permission issue... Is the port 9997 already used ?

Explorer

Port 9997 is exposed from the container.

docker-compose ps:
splunk_splunk-forwarder_1   /sbin/entrypoint.sh start- ...   Up      0.0.0.0:8082->8082/tcp, 8088/tcp, 8089/tcp, 9997/tcp     

But beside that , I don't think its in use - I can't debug it because the container crash after about 30 seconds.

0 Karma

SplunkTrust
SplunkTrust

And im guessing if you try to setup the forwarder alone following this: https://github.com/splunk/docker-splunk-legacy/tree/master/universalforwarder It works, right ?

Explorer

@David, First of all thank you. I'm not familiar with that image - I will try it now and Update.

0 Karma

Explorer

@DavidHourani , it seems like the exact docker image - "splunk/universalforwarder".

0 Karma

SplunkTrust
SplunkTrust

yeah it should be, try calling it alone, see if you have this problem. There's a compose example in the link.

0 Karma

Explorer

This is quite embarrassing..
Copy and pasted the compose.yaml file , rebuilt the image and Eeecuted it along - Still failing:

              Name                             Command               State    Ports
-----------------------------------------------------------------------------------
splunk_splunkuniversalforwarder_1   /sbin/entrypoint.sh start- ...   Exit 1   
0 Karma

SplunkTrust
SplunkTrust

This block is not working ??

volumes:
  opt-splunk-etc:
  opt-splunk-var:

services:
  splunkuniversalforwarder:

    hostname: splunkuniversalforwarder
    image: splunk/universalforwarder:7.0.3
    environment: SPLUNK_START_ARGS: --accept-license
    volumes:
      - opt-splunk-etc:/opt/splunk/etc
      - opt-splunk-var:/opt/splunk/var
    ports:
      - "8000:8000"
      - "9997:9997"
      - "8088:8088"
      - "1514:1514"
0 Karma

Explorer

With the addition of the first line from the example:

version: '3'

No, it is not working, I'm sorry.

0 Karma

Splunk Employee
Splunk Employee

supported containers are 7.2+. Anything earlier I recommend you dont use!

0 Karma

Explorer

you mean an image of splunk/universalforwarder:7.2 and above?

0 Karma

Splunk Employee
Splunk Employee

correct. Tags 7.2+

0 Karma