topic Re: Container name for host using splunk universal forwarder in Deployment Architecture

Container name for host using splunk universal forwarder

ppierson — Sun, 06 Nov 2016 05:04:17 GMT

I am using the universal forwarder to collect logs from docker hosts however when i see the docker containers it has collected logs from it only shows the shortened version of their docker container id. The universal forwarder is listed correctly but the rest are not. Does anyone know how to correct this?

The output looks like so:

Host Count Last Update
0c3344bac2fe Quick Report 76 11/6/16 4:55:30.000 AM
3708dc8f8aff Quick Report 4 11/6/16 4:55:30.000 AM
9efb179e4653 Quick Report 13 11/6/16 4:55:30.000 AM
a043ad123e05 Quick Report 5 11/6/16 4:55:30.000 AM
dcbb531a48a0 Quick Report 166 11/6/16 4:55:30.000 AM
e3a71cd5188e Quick Report 34 11/6/16 4:55:30.000 AM
f93768a45cba Quick Report 84 11/6/16 4:55:30.000 AM
splunkuniversalforwarder Quick Report 5,831 11/6/16 5:05:15.000 AM

As you can see above only splunkuniversalforwarder is named correctly.

Re: Container name for host using splunk universal forwarder

jkat54 — Tue, 08 Nov 2016 14:43:57 GMT

Have you tried editing server.conf on each of the containers and restarting splunk on them?

http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Serverconf

[general]
serverName = <ASCII string>
* The name used to identify this Splunk instance for features such as
  distributed search.
* Defaults to <hostname>-<user running splunk>.
* Shall not be an empty string
* May contain environment variables
* After any environment variables have been expanded, the server name
  (if not an IPv6 address) can only contain letters, numbers, underscores,
  dots, and dashes; and it must start with a letter, number, or an
  underscore.

Re: Container name for host using splunk universal forwarder

ppierson — Tue, 08 Nov 2016 15:02:57 GMT

Splunk universal forwarder (SUF) isnt installed on each container. SUF is running as a container and is collecting the logs of each container from the stdout log of each container.

Re: Container name for host using splunk universal forwarder

jkat54 — Tue, 08 Nov 2016 15:22:44 GMT

Can you modify your container scripts/images to modify the server.conf upon launch?

Re: Container name for host using splunk universal forwarder

ppierson — Tue, 08 Nov 2016 15:25:37 GMT

Not really sure what you mean? the containers are properly named. SUF doesnt look at that though because it is pulling the logs that docker posts in json format on the host.

Re: Container name for host using splunk universal forwarder

jkat54 — Tue, 08 Nov 2016 15:51:09 GMT

O. I. C. I misread your reply.

When you pull data into splunk you can specify the host names. How are you getting the docker logs into Splunk?

Re: Container name for host using splunk universal forwarder

ppierson — Tue, 08 Nov 2016 16:00:49 GMT

from what I understand (again I am fairly new to Splunk) Splunk Universal Forwarder is pulling the docker logs via the location that docker saves them to on the host and pushes them to Splunk.

Re: Container name for host using splunk universal forwarder

jkat54 — Tue, 08 Nov 2016 16:22:35 GMT

Ok so then you should have an inputs.conf somewhere that mentions the log files path. Can you find that and post it here please?

An easy way to get it is using the btool command:

/opt/splunkforwarder/bin/splunk btool inputs list --debug

Re: Container name for host using splunk universal forwarder

ppierson — Tue, 08 Nov 2016 18:19:33 GMT

Hope this helps? Thanks for the input.

Added to pastebin because it was too long for the amount of characters available. http://pastebin.com/C30iLrDv

Re: Container name for host using splunk universal forwarder

jkat54 — Tue, 08 Nov 2016 18:34:51 GMT

Line 64 on you pastebin is an inputs.conf stanza in your docker ta app. This is what is "figuring" out the container name and it is a regex that's being applied to the path of the docker logs to extract the instance names.

If you'll open that docker log path and see if they contain the full name or the shortened name, then we can proceed with a solution. If the full name is listed, can you provide examples so we can give you a regex that should work. If the short name is listed then we will need to make your logging more verbose or set different logging options within docker.

The path is after monitor:// on line 59 in your paste. Where you see the * is where the short or long instance names should appear.

Re: Container name for host using splunk universal forwarder

ppierson — Tue, 08 Nov 2016 18:42:10 GMT

Oh very interesting. I took a look at /host/containers/

root@splunkuniversalforwarder:/opt/splunk# ls -lah /host/containers/
total 36K
drwx------. 9 root root 4.0K Nov 8 18:14 .
drwxr-xr-x 3 root root 24 Nov 8 18:14 ..
drwx------ 3 root root 4.0K Nov 8 18:37 0bda263c181e2d5b36d4a66c97dbf227f0aeb8f5708bc75469c5776e7cb1ab0d
drwx------ 3 root root 4.0K Nov 8 18:37 0fb22ca44d6a781dd3005d3972f1986928266eb0d0d5584f835d11799d220c27
drwx------ 3 root root 4.0K Nov 8 18:37 526edb07ef9255307dba4b841891f33125f4d3fd81667565229232a5debc901f
drwx------ 3 root root 4.0K Nov 8 18:37 710257a95c0c6c71ef88f508d0110211de86908b4886ca47828664ca91d2021b
drwx------ 3 root root 4.0K Nov 8 18:37 738ea4b1ca161c6c2f80698c8249e31565f6fed4e47da163c47ec68c25bc3c07
drwx------ 3 root root 4.0K Nov 8 18:37 753b3561356f0475216da4221797f038a1b3a79e340e260fbf7b390d166b1e3c
drwx------ 3 root root 4.0K Nov 8 18:37 c76778c03115dc5b3ca350729b8a137b33d636ee47366061d3c3acf3a3631f57

And then picked one and ran:
ls -lah /host/containers/c76778c03115dc5b3ca350729b8a137b33d636ee47366061d3c3acf3a3631f57/

which returned:

total 40K
drwx------ 3 root root 4.0K Nov 8 18:37 .
drwx------. 9 root root 4.0K Nov 8 18:14 ..
-rw-r----- 1 root root 0 Nov 8 18:14 c76778c03115dc5b3ca350729b8a137b33d636ee47366061d3c3acf3a3631f57-json.log
-rw-rw-rw- 1 root root 3.3K Nov 8 18:37 config.v2.json
-rw-rw-rw- 1 root root 1.1K Nov 8 18:37 hostconfig.json
-rw-r--r-- 1 root root 13 Nov 8 18:37 hostname
-rw-r--r-- 1 root root 150 Nov 8 18:37 hosts
-rw-r--r-- 1 root root 76 Nov 8 18:37 resolv.conf
-rw-r--r-- 1 root root 71 Nov 8 18:37 resolv.conf.hash
drwx------ 2 root root 4.0K Nov 8 18:14 shm

Lastly I cat'd the hostname file in that folder and it shows the 12 digit container id c76778c03115

Re: Container name for host using splunk universal forwarder

ppierson — Tue, 08 Nov 2016 22:37:18 GMT

It looks like if I run the regex ("(Name)":"((\"|[^"])*)") on the config.v2.json file it name is listed but I now have to figure out how to get splunk universal forwarder to use that instead.

Re: Container name for host using splunk universal forwarder

jkat54 — Tue, 08 Nov 2016 23:02:24 GMT

It'll be easy. On line 64 in your pastebin. Find that inputs.conf in the docker ta app... And change the {12} to {64}. Or however many characters there are in those long container names. Then restart the universal forwarder.