topic Re: Sourcetypes with Docker and HTTP Event Collector in Getting Data In

Sourcetypes with Docker and HTTP Event Collector

sloshburch — Tue, 29 Sep 2020 11:06:33 GMT

(Trying to pull a few similar discussions together and recorded for posterity)

Challenge

The current Docker Logging Driver for Splunk sends HTTP events to Splunk as single-line JSON events because Docker treats each line flushed to stdout / stderror as a separate event (thanks @halr9000). Example:

Notice that those ideally are our access_combined data but since the data is json, we can't get all the field parsing that comes with the out-of-the-box access_combined. This means that you're in a pickle trying to sourcetype the line payload.
Multi-line events, like java stack traces, arrive line by line with this implementation because the connection is not held open until the the event finishes (thanks @Michael Wilde).

How can this be addressed to enjoy the power of my existing sourcetypes with this HTTP Event Collector payload from Docker?

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Wed, 21 Sep 2016 18:10:52 GMT

Solution

The strongest solution is in the works! That is for the Docker Logging Driver for Splunk to transmit HTTP Event Collector in raw mode (rather than json), so the events won’t get surrounded by JSON and our normal field extraction stuff will work.
Our yogi @Michael Wilde has been tracking a PR with Docker for specifically this. If and when that's implemented, I hope to update this accordingly.

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Wed, 21 Sep 2016 18:17:35 GMT

Workaround(s)

Whilst we wait for the ideal solution, here's some workarounds to consider:

Traditional UF Monitor: Have the container write its logs to a volume on the host and use a universal forwarder to pick them up from the host and move them to the indexer. Keep in mind that it's not a bad idea to have a forwarder on the host anyway so you can see things outside of the containers. Again, thank you to @Michael Wilde for this premise!
Sourcetype Override: Use a props and transforms to override the sourcetype to rip out the line's payload (similar to this) and rename to the desired sourcetype. With a sourcetype override, the field parsing of the sourcetype name is what is used.

Other ideas? Post 'em up!

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Wed, 21 Sep 2016 19:52:34 GMT

From @dgladkikh: Raw format has been merged to master https://github.com/docker/docker/pull/25786

So it should be available in 1.13 and it is possible to try it with experimental docker or custom build from master

Re: Sourcetypes with Docker and HTTP Event Collector

rarsan_splunk — Wed, 21 Sep 2016 20:37:19 GMT

Just to clarify: this solution solves challenge 1, not 2. Multi-line events like stack traces are still not handled properly as stderr/stdout streams from different containers are interleaved as they are aggregated by Docker logging driver.

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Wed, 21 Sep 2016 21:17:38 GMT

Cross reference to this thread: https://answers.splunk.com/answers/390219/how-to-parse-docker-logs-with-multiple-events-from.html

Re: Sourcetypes with Docker and HTTP Event Collector

dsmc_adv — Thu, 17 Nov 2016 15:42:01 GMT

Keep in mind that that raw events are only supported in Splunk 6.4 and onwards

http://dev.splunk.com/view/event-collector/SP-CAAAE8Y

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Thu, 17 Nov 2016 22:17:07 GMT

Yes! Thanks for adding that info!

Re: Sourcetypes with Docker and HTTP Event Collector

bhavesh91 — Thu, 01 Dec 2016 09:56:48 GMT

How can the fields which are separated by colon like “line” , “tag” and “source” be extracted automatically on source=http:docker for Docker logs while using Http Event Collector , also if the docker logs have the Key Value in the logs how can those appear as fields in Splunk?

For example the log has the following :

{ [-]
line: 2016-11-14 15:22:03,779; [LOG=debug, NAME=bhav, TIME=1,MSG=Goodday, CLIENT=127.0.0.1]
source: stdout
tag: abc02be1be4e
}

I need to see line , source and tag as fields , along with that KV pair should also showup fields like LOG, NAME, MSG and CLIENT .

Can this be done if so how ? We would want a permanent solution so that it can be applied Enterprise wise.

Re: Sourcetypes with Docker and HTTP Event Collector

bhavesh91 — Thu, 01 Dec 2016 09:56:48 GMT

For example the log has the following :

{ [-] 
   line: 2016-11-14 15:22:03,779; [LOG=debug, NAME=bhav, TIME=1,MSG=Goodday, CLIENT=127.0.0.1] 
   source: stdout
   tag: abc02be1be4e 
}

I need to see line , source and tag as fields , along with that KV pair should also show up fields like LOG, NAME, MSG and CLIENT .

Can this be done if so how ? We would want a permanent solution so that it can be applied Enterprise wise.

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Fri, 02 Dec 2016 13:23:40 GMT

(Sounds like this is in regards to using the log driver so I've moved this comment to that solution rather than the answer related to more traditional approaches)

Odd that those fields are not already parsed for you. Does the sourcetype have a props.conf entry for KV_MODE = json? What is the sourcetype being used and where was it defined (by you or by an app from splunkbase)?

Re: Sourcetypes with Docker and HTTP Event Collector

bhavesh91 — Tue, 29 Sep 2020 12:00:57 GMT

We have been using the default sourcetype - json_no_timestamp which shows up on the Data Inputs -> Http Event Collector.

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Fri, 02 Dec 2016 13:49:34 GMT

I'm not sure if json_no_timestamp is an out-of-the-box sourcetype. What is the value of KV_MODE for that sourcetype (Settings -> Sourcetypes)? In fact, maybe provide a screen shot of that sourectypes definition from the web UI (or btool - but not the conf file).

Re: Sourcetypes with Docker and HTTP Event Collector

bhavesh91 — Sun, 04 Dec 2016 17:11:30 GMT

Also this brings us to a point where we will need to start monitoring the GC(Garbage Collection) and Heap exhaustion in a containers using Splunk - how do we that ?

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Mon, 05 Dec 2016 21:33:33 GMT

Slight update that it sounds like 1.13 is more easily accessible these days and you can start using the new driver: http://blogs.splunk.com/2016/12/01/docker-1-13-with-improved-splunk-logging-driver

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Wed, 07 Dec 2016 22:12:28 GMT

Splunk is a dummy here and just simply accepting data the container sends so you'll need the container to send that data as well. So you'll need to expose data from the GC and heap of the container to Splunk over the same driver.

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Wed, 25 Jan 2017 14:00:04 GMT

Looks like 1.13 just recently came out of beta!

Re: Sourcetypes with Docker and HTTP Event Collector

ErikAulin — Fri, 07 Apr 2017 14:05:55 GMT

Is there any updates to multi-line events. Searched around but this is the closest post that discussed this.

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Mon, 10 Apr 2017 12:58:54 GMT

@Michael Wilde - Is this because Docker still spits out each line individually or has this been adjusted on the docker side so as to send a multiline output as one event?

Re: Sourcetypes with Docker and HTTP Event Collector

sloshburch — Mon, 10 Apr 2017 13:00:16 GMT

@rarsan - Are you sure? I thought the logging driver sends data from the container itself and so different containers send different streams.