HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

[anortrup]

I'm attempting to send a log event from a piece of Go code, but am receiving a connection refused error. If I use curl to send test data, I get an OK status back, but I don't see the data show up in my Splunk instance.

My request data looks like this (formatted pretty)

POST /services/collector HTTP/1.1
Host: input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088
Authorization: Splunk {token}
Content-Type: application/json
{"time":1450588381,"host":"Golang","source":"HTTPSplunkEvent","sourcetype":"Test","index":"tweet_harvest","event":"This is a test"}

My error comes back as:

--- FAIL: TestSendEvent (0.51s)
event_test.go:22: Error sending event: "Post https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector: remote error: handshake failure"

My code looks like this:

import (
    "bytes"
    "crypto/tls"
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "net/http/httputil"
    "strconv"
)

//Event is a struct that holds all data to be sent to a Splunk HTTP logging
//endpoint
type Event struct {
    Time       int64       `json:"time"`
    Host       string      `json:"host"`
    Source     string      `json:"source"`
    Sourcetype string      `json:"sourcetype"`
    Index      string      `json:"index"`
    Event      interface{} `json:"event"`
}

//Send the event to the specified Splunk Server
func (e Event) Send(destination string, token string, disableCertValidation bool) error {

    //Ensure we have all of the values for the event
    if e.Time == 0 || e.Host == "" || e.Source == "" || e.Sourcetype == "" || e.Index == "" || e.Event == nil {
        return errors.New("All fields in Event must have a value")
    }

    //Create a byte array with the data
    b, err := json.Marshal(e)

    //Create client and request
    client := &http.Client{}
    request, err := http.NewRequest("POST", destination, bytes.NewBuffer(b))
    if err != nil {
        return err
    }

    //If we are working with a trial account the certificate will be self signed so we want to ignore certificate verification
    if disableCertValidation {
        tr := &http.Transport{
            TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        }
        client.Transport = tr
    }

    header := http.Header{}
    header.Add("Authorization", "Splunk "+token)
    header.Set("Content-Type", "application/json")
    request.Header = header

    resp, err := client.Do(request)
    if err != nil {
        dump, _ := httputil.DumpRequest(request, true)
        fmt.Printf("Dumping request: %s\n", dump)
        return err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        dump, err := httputil.DumpResponse(resp, true)
        fmt.Printf("RESPONSE: %q\n", dump)
        return err
    }

    //Any code other than 200 is an
    var splunkResponse Response
    decoder := json.NewDecoder(resp.Body)
    err = decoder.Decode(&splunkResponse)

    if err != nil {
        fmt.Printf("Error decoding resposne\n")
        return err
    }

    if splunkResponse.Code != SplunkResponseOK {

        fmt.Printf("Splunk Response code not OK\n")
        return errors.New(strconv.Itoa(splunkResponse.Code) + ": " + splunkResponse.Text)
    }

    //Everything is fine return nil error
    return nil

}

And my test code:

import (
    "testing"
    "time"
)

func TestSendEvent(t *testing.T) {
    event := &Event{
        Time:       time.Now().Unix(),
        Index:      "tweet_harvest",
        Source:     "HTTPSplunkEvent",
        Sourcetype: "Test",
        Event:      "This is a test",
        Host:       "Golang",
    }

    err := event.Send("https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector",
        "8CEB3C52-47B9-451E-81A9-4E45A299D41C", true)

    if err != nil {
        t.Fatalf("Error sending event: %#v\n", err.Error())
    }
}

[anortrup_splunk]

Updated answer

This is no longer an issue in the latest version of Go. crypto/tls now supports the ECDH-ECDSA-AES256-GCM-SHA384 certificates used by Splunk Cloud trial's HTTP event collector.

I've tested this using this library which provides an io.Writer that writes to HTTP event collector.

[skazakov]

"The main issue is the type of ECC cert we used is not compatible with several of these stacks.
It has still not been fixed, but it will be soon / our cloud team is working on it."
https://answers.splunk.com/comments/371925/view.html

Is there any update on this issue? We trying to evaluate Trial Cloud Splunk with Windows WinInet client but failed due to that error.,"The main issue is the type of ECC cert we used is not compatible with several of these stacks.
It has still not been fixed, but it will be soon / our cloud team is working on it."
https://answers.splunk.com/comments/371925/view.html

Is there any update on this issue? We trying to evaluate Trial Cloud Splunk with Windows WinInet client but fails due to that error.

[anortrup · Accepted solution]

So I think that I've narrowed this down to Go doesn't support the self signed certificate used by the trial version of Splunk Cloud (ECDH-ECDSA-AES256-GCM-SHA384).

The list of supported cipher suites for go is:

    TLS_RSA_WITH_RC4_128_SHA                uint16 = 0x0005
    TLS_RSA_WITH_3DES_EDE_CBC_SHA           uint16 = 0x000a
    TLS_RSA_WITH_AES_128_CBC_SHA            uint16 = 0x002f
    TLS_RSA_WITH_AES_256_CBC_SHA            uint16 = 0x0035
    TLS_ECDHE_ECDSA_WITH_RC4_128_SHA        uint16 = 0xc007
    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA    uint16 = 0xc009
    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA    uint16 = 0xc00a
    TLS_ECDHE_RSA_WITH_RC4_128_SHA          uint16 = 0xc011
    TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA     uint16 = 0xc012
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA      uint16 = 0xc013
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA      uint16 = 0xc014
    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256   uint16 = 0xc02f
    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 uint16 = 0xc02b
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   uint16 = 0xc030
    TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c

so I'm going to have to find an alternative way to do this. That said, I suspect that my code would work if I had a paid instance of Splunk with real certificates, which is more than I can commit to for an experimental project.

[jkat54]

I agree. Sorry we couldn't make it work. Probably works fine with Splunk enterprise if you want to test. Also I covered your comment to the answer. Please mark it as such if you can.

[khourihan_splun]

so for curl, since the certificates are self-signed throw the -k flag

curl -k https://http-inputs-stackname.splunkcloud.com/services/collector/event -H "Authorization: Splunk [your token]" -d '{"event": "Tie me kangaroo down, sport"}'

-k, --insecure
(SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections con‐
sidered "insecure" fail unless -k, --insecure is used.

[anortrup]

Curl worked fine, the problem is that the Go doesn't support the non ephemeral key used by the trial certifications.

[gblock_splunk]

@anortrup the cert is set to get updated to one that will be compatible.

[khourihan_splun]

@anortrup From that list of Go supported ciphers, isn't that last one what you need?

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c

I've sent a note over to our CloudOps guys to see if they can help you. Feel free to email me and I can work on this with you. kyle@splunk.com

[gblock_splunk]

@anortrup this works in a Splunk clustered installation, but not in trial / single instance yet. It is a known issue that we are working on.

[circleup]

Another disappointed customer here who just saw Amazon EC2 Container Service supports Splunk, but can't use it because we have a paid single instance :(.

Any updates since 6 months ago?

[gblock_splunk]

Hi @circleup, unfortunately not yet. I am trying to get this work done. I apologize it is taking so long.

[circleup]

Thanks @gblock_splunk, but can you give any indication what timeframe you are trying to get this work done? Days? Weeks? Months?

We just started using Splunk Light Cloud with the hopes of simplifying our infrastructure. But this is turning into a nightmare. Support has basically told me that unless we upgrade to >25GB plan (too expensive to even show on the site's pricing...), "there is no other way to may [sic] this work."

[gdiazlo_bbva]

We're working on a POC to evaluate splunk cloud. We wanted to integrate docker containers with the splunk driver, but it does not work with splunk cloud trial for the very same reason. do you have an aproximate date for this issue to be solved? can we ask for a different trial for our poc so we can evaluate integrations and funcionality with our set up?

[ncrisler]

Do you know if this has been fix or if there is a workaround?

[jkat54]

Is the port the correct splunkd port? By default its 8089 not 8088 as you have written in your script.

https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector

should probably be this instead:

https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8089/services/collector

[netrc]

port 8088 worked for me, but note the documentation for the "curl" example - it uses the "-k" option to allow "insecure" SSL connections.

I got nodeJS to work by setting the "rejectUnauhtorized: false" option to the https.request function.

Nevertheless, I've still seen Splunk return 200 (OK) to the https POST but still drop data.

[gblock_splunk]

@netrc what do you mean it is dropping data?

In terms of the self-signed cert, it might work for node, but it doesn't work for several other stacks we tested namely .NET and Java. The main issue is the type of ECC cert we used is not compatible with several of these stacks.

It has still not been fixed, but it will be soon / our cloud team is working on it.

[barona]

Any news on this issue? I think it's also the root cause for docker splunk logging driver problems.

[anortrup]

The documentation for event collector uses 8088 not 8089. I can use 8089, and I change from getting Connection Refused to 401 Not Properly Authenticated. Not sure if that is progress or not.

[jkat54]

Yeah, I made comment on the documentation page asking if it should be 8088 or 8089. I'm certain its 8089 because it's the REST api and the REST API is port 8089 by default.

SO now you have auth failed, you need to add your credentials to your script.

https://golang.org/pkg/net/url/

The url package seems to support username and password. You might also cheat by putting username and password in the uri using http lib you're using ... http://user:pass@host:8089/endpoints...

This tutorial tells you how to use curl to auth with the restapi if you look around the interwebs youll find other examples:

http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTTUT/RESTconfigurations

I always start with curl and make it work there first. Once i have it working with curl, i script it otherwise.

I just feel it's faster that way. Of course curl is a linux command... i use advanced rest api addon for google chrome when in windows.