Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

[particlebrandon]

I am using Splunk Cloud with the free trial period right now. I need to verify that we are able to use Splunk Cloud with Docker log-driver before we actually move forward with Splunk long-term. I turned on the HTTP Event Collector in Splunk, but I am not able to pass logs via the Docker log-driver options even with splunk-insecureskipverify set to true. See below.

docker run --log-driver=splunk --log-opt splunk-token=C041DEEB-XXXX-XXX-9F5F-3XXXXXXXXXD1C --log-opt splunk-url=https://input-prd-p-5XXXXXXXXX.cloud.splunk.com:8088 --log-opt splunk-insecureskipverify=true hello-world
docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.

Although I did verify the the HTTP event collector is working with the curl command provided. Although that includes /services/collector in the URL, when that is passed to docker run command, it errors out not expecting it to include the full URI.

[gblock_splunk]

Hi @micahhausler.

This is a known issue, it is due to the type of ECC cert that we have used for self-service Splunk Cloud. The Docker driver works fine for our managed cloud but not for self-service. There is a plan to fix this.

[limbirdj]

What is the time frame on the plan to fix this?

[micahhausler]

Would you like me to select this as the accepted answer or wait until the cert type is corrected?

[vvelpuri]

I am not using self service

[vvelpuri]

Hi i am trying to run docker with splunk logging driver . I am using splunk cloud managed service . I am receiving the below error any time did you face this error.

$sudo docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=xxxxxxxxxxx --log-opt splunk-url=https://http-inputs-ccccccc.splunkcloud.com/services/collector/event:8088 --log-opt splunk-insecureskipverify=true nginx

docker: Error response from daemon: Failed to initialize logging driver: splunk: expected format schema://dns_name_or_ip:port for splunk-url.

[rdimri_splunk]

have you tried removing '/services/collector/event' from splunk-url, atleast that is what the error says?

[vvelpuri]

I tried the url in broweser its giving https://http-inputs-xxxxxx.splunkcloud.com/

Its giving
Not Found

The requested URL was not found on this server.

[rdimri_splunk]

If you look at docker code for splunk driver:

https://github.com/docker/docker/blob/master/daemon/logger/splunk/splunk.go#L569

it appends 'services/collector/event' to the configured url.
So when you use browser append that path to splunk host.
When you use splunk driver dont append that.

[rdimri_splunk]

On Splunk enterprise I was able to set docker to data to Splunk, using HTTP Event Collector, running docker with same command as you have specified in the question. Except that I also ensured on splunk Enterprise indexer acknowledgement was disabled.

I suspect that there could be two problems that you could be running into.
1) You may not have opened a ticket with Splunk support so that they can do the needful to allow incoming HTTP requests on the port (firewall may be blocking the port).

https://answers.splunk.com/answers/432236/i-am-a-splunk-cloud-customer-i-want-to-use-the-htt.html

2) Disable indexer acknowledgement.

3) If you have done 1 and 2 above run docker with --debug flag. and paste the output here. We can take a look at it.
4) I am not sure if Splunk cloud will actually open a support ticket for free trial version (I could be wrong). To solve the chicken and egg problem (having to buy license in order to evaluate if this works for you or not) In that case I would recommend downloading a splunk enterprise version and see if it meets your needs, if it does, buy it and then you can request all the support that you want.

[micahhausler]

I don't think that is the case, as I was using Splunk Cloud self-service and I was able to curl requests to the provided domain and get responses

[rdimri_splunk]

What error do you get when you run docker with --debug options ?

[particlebrandon]

Just to piggy back on @micahhaulser i was also able to curl directly to the HEC end point also with no problems. HEC appears to be working just not thru the docker log-driver.

Also yes you can not open an support ticket with the free trail so stuck in an chicken vs egg problem there. Also not sure why using Splunk Enterprise would help me here, considering Splunk Cloud would be an better fit for us in terms of log use vs cost of actually deploying Splunk Enterprise.

[rdimri_splunk]

Using splunk enterprise for this specific case breaks the chicken->egg cycle, if you can get it to work with a trial splunk enterprise(you have more control over what you want to do), then you can go and use splunk cloud (cause it should work there too). Most of the things that work with splunk enterprise will work with cloud too.
I was asking to do this to know if HTTP Event Collector has been configured correctly. Seems like it is cause you can make curl requests and they get indexed.

I am going to dig on this more on docker side, i tried with docker version 1.12.1, I think since its a minor version change from what you ran, the issue might be something else. I will comeback once I have an answer.

[rdimri_splunk]

Okay this is not supported for splunk cloud trial, the problem is that splunk cloud trial uses ECDSA, which although being strong, is not very widely supported. Docker fails handshake because of that.
What you can do is get a real cloud account and then ask for a different certificate.
In addition to golang it also affects .NET.

[particlebrandon]

Good to know. Can we also get an different certificate with Splunk Light Cloud Service?

I find the overall naming convention used by Splunk to be very confusing.

Thanks again for the information, very helpful.

[rdimri_splunk]

Not sure, youll have to talk to someone in marketing probably. Or post a different question.

[barona]

We are using Splunk Light Cloud Service and are also having similar handshake failure problems with docker splunk logging driver. @rdimri: Do you mean that the problem could be solved by requesting new certificates for our Splunk Light? If so, how could that be done? Thank you.

[rdimri_splunk]

@barona, could you please post another question with this specific question, it will be nice if some one from splunk-cloud can authoritatively answer that. There are multiple types of cloud offerings and I am not sure what is the right bucket for this specific request.
I can however confirm that we are working on resolving this issue.

[micahhausler]

DEBU[0000] Trusting certs with subjects: [010U

micahhausler] 
DEBU[0000] Corrupted prefix: []                         
DEBU[0000] [hijack] End of stdout                       
docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.

[micahhausler]

$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:44:17 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      linux/amd64