topic Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure." in Getting Data In

Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

particlebrandon — Mon, 29 Aug 2016 23:14:56 GMT

I am using Splunk Cloud with the free trial period right now. I need to verify that we are able to use Splunk Cloud with Docker log-driver before we actually move forward with Splunk long-term. I turned on the HTTP Event Collector in Splunk, but I am not able to pass logs via the Docker log-driver options even with splunk-insecureskipverify set to true. See below.

docker run --log-driver=splunk --log-opt splunk-token=C041DEEB-XXXX-XXX-9F5F-3XXXXXXXXXD1C --log-opt splunk-url=https://input-prd-p-5XXXXXXXXX.cloud.splunk.com:8088 --log-opt splunk-insecureskipverify=true hello-world
docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.

Although I did verify the the HTTP event collector is working with the curl command provided. Although that includes /services/collector in the URL, when that is passed to docker run command, it errors out not expecting it to include the full URI.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

barona — Thu, 08 Sep 2016 13:41:52 GMT

Did you manage to get docker splunk logging driver work? I'm having exactly the same problem.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

micahhausler — Thu, 08 Sep 2016 13:43:22 GMT

I gave up and went with Fluentd + AWS Cloudwatch Logs + AWS Elasticsearch. Its a breeze to set up

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

particlebrandon — Thu, 08 Sep 2016 16:18:24 GMT

I am ready to give up also, debating on moving back to ELK personally. There was an posting which 1 someone from Splunk mentioned that self-service certs are not supported in golang. I was confused on if that was in Splunk Light or Splunk Cloud or if there was any difference.

At this point I assume there is not any difference and currently Splunk Light/Cloud does not support the docker log-driver.

Sucks because Splunk would have been an perfect fit for me with our logging needs.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Thu, 08 Sep 2016 18:57:36 GMT

On Splunk enterprise I was able to set docker to data to Splunk, using HTTP Event Collector, running docker with same command as you have specified in the question. Except that I also ensured on splunk Enterprise indexer acknowledgement was disabled.

I suspect that there could be two problems that you could be running into.
1) You may not have opened a ticket with Splunk support so that they can do the needful to allow incoming HTTP requests on the port (firewall may be blocking the port).

https://answers.splunk.com/answers/432236/i-am-a-splunk-cloud-customer-i-want-to-use-the-htt.html

2) Disable indexer acknowledgement.

3) If you have done 1 and 2 above run docker with --debug flag. and paste the output here. We can take a look at it.
4) I am not sure if Splunk cloud will actually open a support ticket for free trial version (I could be wrong). To solve the chicken and egg problem (having to buy license in order to evaluate if this works for you or not) In that case I would recommend downloading a splunk enterprise version and see if it meets your needs, if it does, buy it and then you can request all the support that you want.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

micahhausler — Thu, 08 Sep 2016 19:01:59 GMT

I don't think that is the case, as I was using Splunk Cloud self-service and I was able to curl requests to the provided domain and get responses

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Thu, 08 Sep 2016 19:06:29 GMT

What error do you get when you run docker with --debug options ?

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

micahhausler — Thu, 08 Sep 2016 19:11:37 GMT

DEBU[0000] Trusting certs with subjects: [010U

micahhausler] 
DEBU[0000] Corrupted prefix: []                         
DEBU[0000] [hijack] End of stdout                       
docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

micahhausler — Thu, 08 Sep 2016 19:12:05 GMT

$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:44:17 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      linux/amd64

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

particlebrandon — Thu, 08 Sep 2016 19:18:31 GMT

Just to piggy back on @micahhaulser i was also able to curl directly to the HEC end point also with no problems. HEC appears to be working just not thru the docker log-driver.

Also yes you can not open an support ticket with the free trail so stuck in an chicken vs egg problem there. Also not sure why using Splunk Enterprise would help me here, considering Splunk Cloud would be an better fit for us in terms of log use vs cost of actually deploying Splunk Enterprise.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Thu, 08 Sep 2016 21:07:45 GMT

Using splunk enterprise for this specific case breaks the chicken->egg cycle, if you can get it to work with a trial splunk enterprise(you have more control over what you want to do), then you can go and use splunk cloud (cause it should work there too). Most of the things that work with splunk enterprise will work with cloud too.
I was asking to do this to know if HTTP Event Collector has been configured correctly. Seems like it is cause you can make curl requests and they get indexed.

I am going to dig on this more on docker side, i tried with docker version 1.12.1, I think since its a minor version change from what you ran, the issue might be something else. I will comeback once I have an answer.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Fri, 09 Sep 2016 17:22:32 GMT

Okay this is not supported for splunk cloud trial, the problem is that splunk cloud trial uses ECDSA, which although being strong, is not very widely supported. Docker fails handshake because of that.
What you can do is get a real cloud account and then ask for a different certificate.
In addition to golang it also affects .NET.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

particlebrandon — Fri, 09 Sep 2016 18:28:32 GMT

Good to know. Can we also get an different certificate with Splunk Light Cloud Service?

I find the overall naming convention used by Splunk to be very confusing.

Thanks again for the information, very helpful.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Fri, 09 Sep 2016 18:39:50 GMT

Not sure, youll have to talk to someone in marketing probably. Or post a different question.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

gblock_splunk — Sat, 10 Sep 2016 05:47:21 GMT

Hi @micahhausler.

This is a known issue, it is due to the type of ECC cert that we have used for self-service Splunk Cloud. The Docker driver works fine for our managed cloud but not for self-service. There is a plan to fix this.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

barona — Thu, 15 Sep 2016 10:37:24 GMT

We are using Splunk Light Cloud Service and are also having similar handshake failure problems with docker splunk logging driver. @rdimri: Do you mean that the problem could be solved by requesting new certificates for our Splunk Light? If so, how could that be done? Thank you.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Fri, 16 Sep 2016 03:36:22 GMT

@barona, could you please post another question with this specific question, it will be nice if some one from splunk-cloud can authoritatively answer that. There are multiple types of cloud offerings and I am not sure what is the right bucket for this specific request.
I can however confirm that we are working on resolving this issue.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

vvelpuri — Tue, 29 Sep 2020 11:07:08 GMT

Hi i am trying to run docker with splunk logging driver . I am using splunk cloud managed service . I am receiving the below error any time did you face this error.

$sudo docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=xxxxxxxxxxx --log-opt splunk-url=https://http-inputs-ccccccc.splunkcloud.com/services/collector/event:8088 --log-opt splunk-insecureskipverify=true nginx

docker: Error response from daemon: Failed to initialize logging driver: splunk: expected format schema://dns_name_or_ip:port for splunk-url.

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

rdimri_splunk — Thu, 22 Sep 2016 14:06:00 GMT

have you tried removing '/services/collector/event' from splunk-url, atleast that is what the error says?

Re: Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

micahhausler — Thu, 22 Sep 2016 14:28:48 GMT

Would you like me to select this as the accepted answer or wait until the cert type is corrected?