Getting Data In
Highlighted

How to parse Docker logs with multiple events from stacktrace?

Explorer

When using the Docker Splunk logging driver to send events into the http collector splunk logs individual logs like this:

{"line":"the message","source":"stdout","tag":"container tag"}

Unfortunately, for stacktraces from tomcat/log4j, it will separate them into multiple log events per line for the stacktrace, bottom line first, like this:

{"line":"\tat java.lang.Thread.run(Thread.java:745)\r","source":"stdout","tag":"33f25cc98f0c"}
{"line":"\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r","source":"stdout","tag":"33f25cc98f0c"}
{"line":"\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r","source":"stdout","tag":"33f25cc98f0c"}

This makes is it nearly impossible to use. Does anybody know a way to either combine them in Splunk or get tomcat to spit them out in a single line?

Hope someone out there are able to help.

Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Engager

Splunk-guys - you really need to answer this....we have the same issue and are just about to purchase a Splunk Ent. license. But this is a show stopper for sure.....

Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

Whichever Docker log driver you use you'll face the same issue. Docker treats each line flushed to stdout / stderror as a separate event. This is perfectly reasonable behavior as it cannot know what constitutes the start and end of a multi-line log event.

I came to the conclusion the easiest way to address this was at source. Alter your log4j layout to escape the newlines in stack traces.

Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

Cheers @nickperry, that's exactly what I've done now. For anybody interested, it seems the easiest way with log4j is to implement your own ThrowableRenderer, this is what we're using now:

package com.mypackage;

import org.apache.log4j.EnhancedThrowableRenderer;
import org.apache.log4j.spi.ThrowableRenderer;

public class CustomThrowableRenderer implements ThrowableRenderer {

    EnhancedThrowableRenderer renderer = new EnhancedThrowableRenderer();

    @Override
    public String[] doRender(Throwable t) {
        String[] output = new String[1];

        for (String line : renderer.doRender(t)) {
            output[0] += line.replace("\n", "") + "\\n";
        }

        return output;
    }
}

and add it to log4j custom as log4j.throwableRenderer=com.mypackage.CustomThrowableRenderer. Should be possible then to get splunk to interpret '\n' as a newline.

Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

How do you feel about the Search and Report statements being wrapped in JSON? (read this in a scientifically interested tone, not a judging one 😉 )

0 Karma
Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

Can you clarify what you mean?

0 Karma
Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

Cool. Thanks for the code. We have some applications that should benefit from this. Will try it shortly.

0 Karma
Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

Well, if the interest is in the "line", I personally find it annoying that it is wrapped within a json object, instead of just outputting the line, when I make my searches.

0 Karma
Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

Just use | spath path=line output=_raw in your searches

0 Karma
Highlighted

Re: How to parse Docker logs with multiple events from stacktrace?

Explorer

But the indexer is unable to do that, and parse only the "line" segment, right?

0 Karma