Getting Data In

Splunk Cloud & HTTP Event Collector: Docker log-driver error "Failed to initialize logging driver: remote error: handshake failure."

I am using Splunk Cloud with the free trial period right now. I need to verify that we are able to use Splunk Cloud with Docker log-driver before we actually move forward with Splunk long-term. I turned on the HTTP Event Collector in Splunk, but I am not able to pass logs via the Docker log-driver options even with splunk-insecureskipverify set to true. See below.

docker run --log-driver=splunk --log-opt splunk-token=C041DEEB-XXXX-XXX-9F5F-3XXXXXXXXXD1C --log-opt splunk-url=https://input-prd-p-5XXXXXXXXX.cloud.splunk.com:8088 --log-opt splunk-insecureskipverify=true hello-world
docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.

Although I did verify the the HTTP event collector is working with the curl command provided. Although that includes /services/collector in the URL, when that is passed to docker run command, it errors out not expecting it to include the full URI.

Splunk Employee
Splunk Employee

Hi @micahhausler.

This is a known issue, it is due to the type of ECC cert that we have used for self-service Splunk Cloud. The Docker driver works fine for our managed cloud but not for self-service. There is a plan to fix this.

New Member

What is the time frame on the plan to fix this?

0 Karma

Engager

Would you like me to select this as the accepted answer or wait until the cert type is corrected?

0 Karma

Explorer

I am not using self service

0 Karma

Explorer

Hi i am trying to run docker with splunk logging driver . I am using splunk cloud managed service . I am receiving the below error any time did you face this error.

$sudo docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=xxxxxxxxxxx --log-opt splunk-url=https://http-inputs-ccccccc.splunkcloud.com/services/collector/event:8088 --log-opt splunk-insecureskipverify=true nginx

docker: Error response from daemon: Failed to initialize logging driver: splunk: expected format schema://dns_name_or_ip:port for splunk-url.

0 Karma

Splunk Employee
Splunk Employee

have you tried removing '/services/collector/event' from splunk-url, atleast that is what the error says?

0 Karma

Explorer

I tried the url in broweser its giving https://http-inputs-xxxxxx.splunkcloud.com/

Its giving
Not Found

The requested URL was not found on this server.

0 Karma

Splunk Employee
Splunk Employee

If you look at docker code for splunk driver:

https://github.com/docker/docker/blob/master/daemon/logger/splunk/splunk.go#L569

it appends 'services/collector/event' to the configured url.
So when you use browser append that path to splunk host.
When you use splunk driver dont append that.

0 Karma

Splunk Employee
Splunk Employee

On Splunk enterprise I was able to set docker to data to Splunk, using HTTP Event Collector, running docker with same command as you have specified in the question. Except that I also ensured on splunk Enterprise indexer acknowledgement was disabled.

I suspect that there could be two problems that you could be running into.
1) You may not have opened a ticket with Splunk support so that they can do the needful to allow incoming HTTP requests on the port (firewall may be blocking the port).

https://answers.splunk.com/answers/432236/i-am-a-splunk-cloud-customer-i-want-to-use-the-htt.html

2) Disable indexer acknowledgement.

3) If you have done 1 and 2 above run docker with --debug flag. and paste the output here. We can take a look at it.
4) I am not sure if Splunk cloud will actually open a support ticket for free trial version (I could be wrong). To solve the chicken and egg problem (having to buy license in order to evaluate if this works for you or not) In that case I would recommend downloading a splunk enterprise version and see if it meets your needs, if it does, buy it and then you can request all the support that you want.

0 Karma

Engager

I don't think that is the case, as I was using Splunk Cloud self-service and I was able to curl requests to the provided domain and get responses

0 Karma

Splunk Employee
Splunk Employee

What error do you get when you run docker with --debug options ?

0 Karma

Just to piggy back on @micahhaulser i was also able to curl directly to the HEC end point also with no problems. HEC appears to be working just not thru the docker log-driver.

Also yes you can not open an support ticket with the free trail so stuck in an chicken vs egg problem there. Also not sure why using Splunk Enterprise would help me here, considering Splunk Cloud would be an better fit for us in terms of log use vs cost of actually deploying Splunk Enterprise.

0 Karma

Splunk Employee
Splunk Employee

Using splunk enterprise for this specific case breaks the chicken->egg cycle, if you can get it to work with a trial splunk enterprise(you have more control over what you want to do), then you can go and use splunk cloud (cause it should work there too). Most of the things that work with splunk enterprise will work with cloud too.
I was asking to do this to know if HTTP Event Collector has been configured correctly. Seems like it is cause you can make curl requests and they get indexed.

I am going to dig on this more on docker side, i tried with docker version 1.12.1, I think since its a minor version change from what you ran, the issue might be something else. I will comeback once I have an answer.

0 Karma

Splunk Employee
Splunk Employee

Okay this is not supported for splunk cloud trial, the problem is that splunk cloud trial uses ECDSA, which although being strong, is not very widely supported. Docker fails handshake because of that.
What you can do is get a real cloud account and then ask for a different certificate.
In addition to golang it also affects .NET.

0 Karma

Good to know. Can we also get an different certificate with Splunk Light Cloud Service?

I find the overall naming convention used by Splunk to be very confusing.

Thanks again for the information, very helpful.

0 Karma

Splunk Employee
Splunk Employee

Not sure, youll have to talk to someone in marketing probably. Or post a different question.

0 Karma

Explorer

We are using Splunk Light Cloud Service and are also having similar handshake failure problems with docker splunk logging driver. @rdimri: Do you mean that the problem could be solved by requesting new certificates for our Splunk Light? If so, how could that be done? Thank you.

0 Karma

Splunk Employee
Splunk Employee

@barona, could you please post another question with this specific question, it will be nice if some one from splunk-cloud can authoritatively answer that. There are multiple types of cloud offerings and I am not sure what is the right bucket for this specific request.
I can however confirm that we are working on resolving this issue.

0 Karma

Engager
DEBU[0000] Trusting certs with subjects: [010U

micahhausler] 
DEBU[0000] Corrupted prefix: []                         
DEBU[0000] [hijack] End of stdout                       
docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.
0 Karma

Engager
$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:44:17 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:20:08 2016
 OS/Arch:      linux/amd64
0 Karma