I've finally found a way around the problem, at least for Splunk.
Note that with this solution, you'll not be able to use any of the added labels that the Splunk Log driver gives you. In my team, we got around this by including all the information within the log statement (through either the MDC or the logging configuration within the service in the container).
Instead of using the Splunk Log driver (which leaves little control about what to do), we used the Syslog driver. We configured the Syslog driver to send its data via UDP to a normal Splunk installation. Remember to setup Splunk to listen for data. This can be done in Settings -> Data Input -> UDP (under Local inputs) and then hit new. Set the port to 1514. If you are running your Splunk in a docker container, make a port binding that maps 514 to 1514 (as 514 is the standard syslog port). Create a new source type in the next screen, and name it something you can remember, as we need to configure it afterwards.
Now, when a log event occurs within your service container, it is logged to syslog which sends it to Splunk. It does however come out with the syslog data (date, tags etc) prepended to your log statement and each line is interpreted as its own event.
When the UDP input has been created, it is time to configure the source type. Go to Settings -> Source Types and search for the source type you've just created.
First, lets remove the syslog data from the log statement. In the Advanced section, click "new setting" at the bottom. Name it SEDCMD-01. This will run an SED command, which allows us to modify the input prior to indexing (based on rules described here). On my unix system the syslog data would look something like this
[{date} something=docker horse=horse]: Start my log statement
To remove this, I set the value of my SEDCMD-01 to "s/.+]: //g" without quotes. This takes everything that ends with ]: and removes it, and as that is what end out syslog statement, we now only have the pure log statement. This also means that any tags that is configured with the Syslog Driver is removed prior to indexing, so keep all the extra information you want within your core log statement.
All of my own log statement start with a date and time like "2016-06-30 08:15:30,423". I use this create an event break point. In the Event Breaks section of the source type, I put in "\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d" without quotes in the pattern box. Doing so creates a BREAK_ONLY_BEFORE in my advanced section (so, the break will be before my date by standard). Note that this is not a line break, but event break. That means it will keep gathering data, until it reaches a new date.
In my experience (running Splunk in a docker container), I would have to restart the container in order for my changes to take effect. Otherwise, you should now be done!
I've attached below some images of my configuration in the source type, and of what a log statement looks like in my Search and Report.
I hope this helps. Please ask more questions if it doesn't, so that we can get more people to use docker and splunk together 🙂
... View more
