Getting Data In

Docker Splunk logging driver per line 4K byte limit.

Path Finder

We are using splunk logging driver to send docker container logs and some containers are sending log messages >4K bytes in a single line (like stack traces). We see them getting truncated in Splunk and want to check if any one aware of where the 4K limit is coming from?

Docker sends all logs to stdout and they get picked by logging driver, I saw in some linux forums that stdout pipe buffer limit is 4K, does anyone have more details on this how to resolve it?

0 Karma
1 Solution

Communicator

Hi @kyaparla

Author of the Splunk Logging Driver here.

The messages are getting split inside the copier in the docker daemon itself. https://github.com/moby/moby/blob/dfc2d62632d32f9d38166ea477f0ca033a5c91c2/daemon/logger/copier.go

Seems like change is coming which will increase the limit much higher (80k) https://github.com/moby/moby/pull/35617

Meanwhile, I would suggest you look at the solutions which we have built for monitoring Docker. It is a certified application https://splunkbase.splunk.com/app/3723/ and our custom built collector for docker (kubernetes and openshift), which can collect logs, deal with multiline events and have a lot of other benefits. You can see a list of benefits compared to the Splunk Logging Driver https://www.outcoldsolutions.com/docs/collectorfordocker/

View solution in original post

0 Karma

Communicator

Hi @kyaparla

Author of the Splunk Logging Driver here.

The messages are getting split inside the copier in the docker daemon itself. https://github.com/moby/moby/blob/dfc2d62632d32f9d38166ea477f0ca033a5c91c2/daemon/logger/copier.go

Seems like change is coming which will increase the limit much higher (80k) https://github.com/moby/moby/pull/35617

Meanwhile, I would suggest you look at the solutions which we have built for monitoring Docker. It is a certified application https://splunkbase.splunk.com/app/3723/ and our custom built collector for docker (kubernetes and openshift), which can collect logs, deal with multiline events and have a lot of other benefits. You can see a list of benefits compared to the Splunk Logging Driver https://www.outcoldsolutions.com/docs/collectorfordocker/

View solution in original post

0 Karma

Path Finder

Thanks @outcoldman, This helps a lot, we will wait new version of docker. Will also look at the app, it seems to resolve many of our issues.

Does the app require any docker volumes to be mounted to read log files?

0 Karma

Path Finder

Hi @outcoldman,

I am just reaching to you, if you are aware of recent updates with splunk logging driver? We are seeing that 4K char limit is not an issue any more when splunk-format is inline, but the raw format still has this issue. Appreciate, if you could share any updates or provide some pointers.

0 Karma

Communicator

I have seen that the limit has been increased for all logging drivers. But not sure, why do you see different behavior on different types of logging formats, as this is handled by the driver itself.
As I have mentioned in my previous reply - you can try our solution, that does not have this problem.

0 Karma

Communicator

@kyaparla, that is correct, you can take a look on the manual how to install the application and our collector https://www.outcoldsolutions.com/docs/monitoring-docker/ , our collector is built on top of json-file driver, so you just map the containers folder and some other for metrics. The best part, that you will be able to use docker logs, while you will be able to forward all logs. Also, in case of network failures, your logs will be kept on disk, so when the network issues will be resolved - you will index all the logs from the right place, without losing any information (at least once delivery).

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!