I am using Splunk Cloud with the free trial period right now. I need to verify that we are able to use Splunk Cloud with Docker log-driver before we actually move forward with Splunk long-term. I turned on the HTTP Event Collector in Splunk, but I am not able to pass logs via the Docker log-driver options even with splunk-insecureskipverify set to true. See below.
docker run --log-driver=splunk --log-opt splunk-token=C041DEEB-XXXX-XXX-9F5F-3XXXXXXXXXD1C --log-opt splunk-url=https://input-prd-p-5XXXXXXXXX.cloud.splunk.com:8088 --log-opt splunk-insecureskipverify=true hello-world docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.
Although I did verify the the HTTP event collector is working with the curl command provided. Although that includes /services/collector in the URL, when that is passed to docker run command, it errors out not expecting it to include the full URI.
This is a known issue, it is due to the type of ECC cert that we have used for self-service Splunk Cloud. The Docker driver works fine for our managed cloud but not for self-service. There is a plan to fix this.
Hi i am trying to run docker with splunk logging driver . I am using splunk cloud managed service . I am receiving the below error any time did you face this error.
$sudo docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=xxxxxxxxxxx --log-opt splunk-url=https://http-inputs-ccccccc.splunkcloud.com/services/collector/event:8088 --log-opt splunk-insecureskipverify=true nginx
docker: Error response from daemon: Failed to initialize logging driver: splunk: expected format schema://dns_name_or_ip:port for splunk-url.
If you look at docker code for splunk driver:
it appends 'services/collector/event' to the configured url.
So when you use browser append that path to splunk host.
When you use splunk driver dont append that.
On Splunk enterprise I was able to set docker to data to Splunk, using HTTP Event Collector, running docker with same command as you have specified in the question. Except that I also ensured on splunk Enterprise indexer acknowledgement was disabled.
I suspect that there could be two problems that you could be running into.
1) You may not have opened a ticket with Splunk support so that they can do the needful to allow incoming HTTP requests on the port (firewall may be blocking the port).
2) Disable indexer acknowledgement.
3) If you have done 1 and 2 above run docker with --debug flag. and paste the output here. We can take a look at it.
4) I am not sure if Splunk cloud will actually open a support ticket for free trial version (I could be wrong). To solve the chicken and egg problem (having to buy license in order to evaluate if this works for you or not) In that case I would recommend downloading a splunk enterprise version and see if it meets your needs, if it does, buy it and then you can request all the support that you want.
Just to piggy back on @micahhaulser i was also able to curl directly to the HEC end point also with no problems. HEC appears to be working just not thru the docker log-driver.
Also yes you can not open an support ticket with the free trail so stuck in an chicken vs egg problem there. Also not sure why using Splunk Enterprise would help me here, considering Splunk Cloud would be an better fit for us in terms of log use vs cost of actually deploying Splunk Enterprise.
Using splunk enterprise for this specific case breaks the chicken->egg cycle, if you can get it to work with a trial splunk enterprise(you have more control over what you want to do), then you can go and use splunk cloud (cause it should work there too). Most of the things that work with splunk enterprise will work with cloud too.
I was asking to do this to know if HTTP Event Collector has been configured correctly. Seems like it is cause you can make curl requests and they get indexed.
I am going to dig on this more on docker side, i tried with docker version 1.12.1, I think since its a minor version change from what you ran, the issue might be something else. I will comeback once I have an answer.
Okay this is not supported for splunk cloud trial, the problem is that splunk cloud trial uses ECDSA, which although being strong, is not very widely supported. Docker fails handshake because of that.
What you can do is get a real cloud account and then ask for a different certificate.
In addition to golang it also affects .NET.
Good to know. Can we also get an different certificate with Splunk Light Cloud Service?
I find the overall naming convention used by Splunk to be very confusing.
Thanks again for the information, very helpful.
We are using Splunk Light Cloud Service and are also having similar handshake failure problems with docker splunk logging driver. @rdimri: Do you mean that the problem could be solved by requesting new certificates for our Splunk Light? If so, how could that be done? Thank you.
@barona, could you please post another question with this specific question, it will be nice if some one from splunk-cloud can authoritatively answer that. There are multiple types of cloud offerings and I am not sure what is the right bucket for this specific request.
I can however confirm that we are working on resolving this issue.
DEBU Trusting certs with subjects: [010U micahhausler] DEBU Corrupted prefix:  DEBU [hijack] End of stdout docker: Error response from daemon: Failed to initialize logging driver: remote error: handshake failure.
$ docker version Client: Version: 1.11.1 API version: 1.23 Go version: go1.5.4 Git commit: 5604cbe Built: Tue Apr 26 23:44:17 2016 OS/Arch: darwin/amd64 Server: Version: 1.11.2 API version: 1.23 Go version: go1.5.4 Git commit: b9f10c9 Built: Wed Jun 1 21:20:08 2016 OS/Arch: linux/amd64