I am using the universal forwarder to collect logs from docker hosts however when i see the docker containers it has collected logs from it only shows the shortened version of their docker container id. The universal forwarder is listed correctly but the rest are not. Does anyone know how to correct this?
The output looks like so:
Host Count Last Update
0c3344bac2fe Quick Report 76 11/6/16 4:55:30.000 AM
3708dc8f8aff Quick Report 4 11/6/16 4:55:30.000 AM
9efb179e4653 Quick Report 13 11/6/16 4:55:30.000 AM
a043ad123e05 Quick Report 5 11/6/16 4:55:30.000 AM
dcbb531a48a0 Quick Report 166 11/6/16 4:55:30.000 AM
e3a71cd5188e Quick Report 34 11/6/16 4:55:30.000 AM
f93768a45cba Quick Report 84 11/6/16 4:55:30.000 AM
splunkuniversalforwarder Quick Report 5,831 11/6/16 5:05:15.000 AM
As you can see above only splunkuniversalforwarder is named correctly.
Have you tried editing server.conf on each of the containers and restarting splunk on them?
[general] serverName = <ASCII string> * The name used to identify this Splunk instance for features such as distributed search. * Defaults to <hostname>-<user running splunk>. * Shall not be an empty string * May contain environment variables * After any environment variables have been expanded, the server name (if not an IPv6 address) can only contain letters, numbers, underscores, dots, and dashes; and it must start with a letter, number, or an underscore.
Splunk universal forwarder (SUF) isnt installed on each container. SUF is running as a container and is collecting the logs of each container from the stdout log of each container.
Not really sure what you mean? the containers are properly named. SUF doesnt look at that though because it is pulling the logs that docker posts in json format on the host.
O. I. C. I misread your reply.
When you pull data into splunk you can specify the host names. How are you getting the docker logs into Splunk?
from what I understand (again I am fairly new to Splunk) Splunk Universal Forwarder is pulling the docker logs via the location that docker saves them to on the host and pushes them to Splunk.
Ok so then you should have an inputs.conf somewhere that mentions the log files path. Can you find that and post it here please?
An easy way to get it is using the btool command:
/opt/splunkforwarder/bin/splunk btool inputs list --debug
Line 64 on you pastebin is an inputs.conf stanza in your docker ta app. This is what is "figuring" out the container name and it is a regex that's being applied to the path of the docker logs to extract the instance names.
If you'll open that docker log path and see if they contain the full name or the shortened name, then we can proceed with a solution. If the full name is listed, can you provide examples so we can give you a regex that should work. If the short name is listed then we will need to make your logging more verbose or set different logging options within docker.
The path is after monitor:// on line 59 in your paste. Where you see the * is where the short or long instance names should appear.