topic Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error? in Getting Data In

HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Sun, 20 Dec 2015 05:31:41 GMT

I'm attempting to send a log event from a piece of Go code, but am receiving a connection refused error. If I use curl to send test data, I get an OK status back, but I don't see the data show up in my Splunk instance.

My request data looks like this (formatted pretty)

POST /services/collector HTTP/1.1
Host: input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088
Authorization: Splunk {token}
Content-Type: application/json
{"time":1450588381,"host":"Golang","source":"HTTPSplunkEvent","sourcetype":"Test","index":"tweet_harvest","event":"This is a test"}

My error comes back as:

--- FAIL: TestSendEvent (0.51s)
event_test.go:22: Error sending event: "Post https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector: remote error: handshake failure"

My code looks like this:

import (
    "bytes"
    "crypto/tls"
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "net/http/httputil"
    "strconv"
)

//Event is a struct that holds all data to be sent to a Splunk HTTP logging
//endpoint
type Event struct {
    Time       int64       `json:"time"`
    Host       string      `json:"host"`
    Source     string      `json:"source"`
    Sourcetype string      `json:"sourcetype"`
    Index      string      `json:"index"`
    Event      interface{} `json:"event"`
}

//Send the event to the specified Splunk Server
func (e Event) Send(destination string, token string, disableCertValidation bool) error {

    //Ensure we have all of the values for the event
    if e.Time == 0 || e.Host == "" || e.Source == "" || e.Sourcetype == "" || e.Index == "" || e.Event == nil {
        return errors.New("All fields in Event must have a value")
    }

    //Create a byte array with the data
    b, err := json.Marshal(e)

    //Create client and request
    client := &http.Client{}
    request, err := http.NewRequest("POST", destination, bytes.NewBuffer(b))
    if err != nil {
        return err
    }

    //If we are working with a trial account the certificate will be self signed so we want to ignore certificate verification
    if disableCertValidation {
        tr := &http.Transport{
            TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        }
        client.Transport = tr
    }

    header := http.Header{}
    header.Add("Authorization", "Splunk "+token)
    header.Set("Content-Type", "application/json")
    request.Header = header

    resp, err := client.Do(request)
    if err != nil {
        dump, _ := httputil.DumpRequest(request, true)
        fmt.Printf("Dumping request: %s\n", dump)
        return err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        dump, err := httputil.DumpResponse(resp, true)
        fmt.Printf("RESPONSE: %q\n", dump)
        return err
    }

    //Any code other than 200 is an
    var splunkResponse Response
    decoder := json.NewDecoder(resp.Body)
    err = decoder.Decode(&splunkResponse)

    if err != nil {
        fmt.Printf("Error decoding resposne\n")
        return err
    }

    if splunkResponse.Code != SplunkResponseOK {

        fmt.Printf("Splunk Response code not OK\n")
        return errors.New(strconv.Itoa(splunkResponse.Code) + ": " + splunkResponse.Text)
    }

    //Everything is fine return nil error
    return nil

}

And my test code:

import (
    "testing"
    "time"
)

func TestSendEvent(t *testing.T) {
    event := &Event{
        Time:       time.Now().Unix(),
        Index:      "tweet_harvest",
        Source:     "HTTPSplunkEvent",
        Sourcetype: "Test",
        Event:      "This is a test",
        Host:       "Golang",
    }

    err := event.Send("https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector",
        "8CEB3C52-47B9-451E-81A9-4E45A299D41C", true)

    if err != nil {
        t.Fatalf("Error sending event: %#v\n", err.Error())
    }
}

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

jkat54 — Sun, 20 Dec 2015 10:00:28 GMT

Is the port the correct splunkd port? By default its 8089 not 8088 as you have written in your script.

https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector

should probably be this instead:

https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8089/services/collector

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Sun, 20 Dec 2015 17:55:01 GMT

The documentation for event collector uses 8088 not 8089. I can use 8089, and I change from getting Connection Refused to 401 Not Properly Authenticated. Not sure if that is progress or not.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

jkat54 — Sun, 20 Dec 2015 22:06:44 GMT

Yeah, I made comment on the documentation page asking if it should be 8088 or 8089. I'm certain its 8089 because it's the REST api and the REST API is port 8089 by default.

SO now you have auth failed, you need to add your credentials to your script.

https://golang.org/pkg/net/url/

The url package seems to support username and password. You might also cheat by putting username and password in the uri using http lib you're using ... http://user:pass@host:8089/endpoints...

This tutorial tells you how to use curl to auth with the restapi if you look around the interwebs youll find other examples:

http://docs.splunk.com/Documentation/Splunk/6.3.2/RESTTUT/RESTconfigurations

I always start with curl and make it work there first. Once i have it working with curl, i script it otherwise.

I just feel it's faster that way. Of course curl is a linux command... i use advanced rest api addon for google chrome when in windows.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

jkat54 — Sun, 20 Dec 2015 22:14:48 GMT

You're so close!

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Mon, 21 Dec 2015 01:11:40 GMT

I see what you are saying about the port difference, and respect that you have way more experience, but I don't think the documentation is in error. If I change the port numbers when using curl I get the correct result with 8088 and an error with 8089.

curl -k https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector/event -H "Authorization: Splunk 8CEB3C52-47B9-451E-81A9-4E45A299D41C" -d '{"event": "hello world"}'

Returns:

{"text":"Success","code":0}

which is what you want.

While:

curl -k https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8089/services/collector/event -H "Authorization: 8CEB3C52-47B9-451E-81A9-4E45A299D41C" -d '{"event": "hello world"}'

returns:

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">Unauthorized</msg>
  </messages>
</response>

I think the difference is that the event collector is using token based authentication rather than the REST client's user/password authentication.

I still have two problems. One, my code still gets a handshake failure with 8088. Two, the event I sent with Curl doesn't show up inside of my Splunk instance.

-Andy

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Mon, 21 Dec 2015 01:15:48 GMT

I fixed the problem with Curl by specifying the index. So I'm down to just the handshake problem with Go.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Mon, 21 Dec 2015 02:36:07 GMT

So I think that I've narrowed this down to Go doesn't support the self signed certificate used by the trial version of Splunk Cloud (ECDH-ECDSA-AES256-GCM-SHA384).

The list of supported cipher suites for go is:

    TLS_RSA_WITH_RC4_128_SHA                uint16 = 0x0005
    TLS_RSA_WITH_3DES_EDE_CBC_SHA           uint16 = 0x000a
    TLS_RSA_WITH_AES_128_CBC_SHA            uint16 = 0x002f
    TLS_RSA_WITH_AES_256_CBC_SHA            uint16 = 0x0035
    TLS_ECDHE_ECDSA_WITH_RC4_128_SHA        uint16 = 0xc007
    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA    uint16 = 0xc009
    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA    uint16 = 0xc00a
    TLS_ECDHE_RSA_WITH_RC4_128_SHA          uint16 = 0xc011
    TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA     uint16 = 0xc012
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA      uint16 = 0xc013
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA      uint16 = 0xc014
    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256   uint16 = 0xc02f
    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 uint16 = 0xc02b
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   uint16 = 0xc030
    TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c

so I'm going to have to find an alternative way to do this. That said, I suspect that my code would work if I had a paid instance of Splunk with real certificates, which is more than I can commit to for an experimental project.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

esix_splunk — Mon, 21 Dec 2015 05:04:11 GMT

The HEC will you 8088 by default. REST API is 8089. In this case, you are using the HEC (HTTP Event Collector) so 8088 is the correct port.

Did you setup the input with authentication from the Splunk Cloud instance?

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Tue, 22 Dec 2015 01:01:37 GMT

Esix,

I created a HEC token as directed by the documentation, but Go doesn't appear to support the cipher suite that the Splunk Cloud Trial comes with, unless their is a way to configure the cipher suite myself in the cloud, I think I'm out of luck for use of HEC.

I think I can drop back and use the REST API instead, but it is going to take some rewriting.

-Andy

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

khourihan_splun — Sun, 17 Jan 2016 19:35:01 GMT

so for curl, since the certificates are self-signed throw the -k flag

curl -k https://http-inputs-stackname.splunkcloud.com/services/collector/event -H "Authorization: Splunk [your token]" -d '{"event": "Tie me kangaroo down, sport"}'

-k, --insecure
(SSL) This option explicitly allows curl to perform "insecure" SSL connections and transfers. All SSL connections are attempted to be made secure by using the CA certificate bundle installed by default. This makes all connections conâ€
sidered "insecure" fail unless -k, --insecure is used.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup — Sun, 17 Jan 2016 19:52:56 GMT

Curl worked fine, the problem is that the Go doesn't support the non ephemeral key used by the trial certifications.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

khourihan_splun — Sun, 17 Jan 2016 20:04:07 GMT

@anortrup From that list of Go supported ciphers, isn't that last one what you need?

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c

I've sent a note over to our CloudOps guys to see if they can help you. Feel free to email me and I can work on this with you. kyle@splunk.com

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

gblock_splunk — Sun, 17 Jan 2016 23:39:35 GMT

@anortrup the cert is set to get updated to one that will be compatible.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

gblock_splunk — Sun, 17 Jan 2016 23:41:58 GMT

@anortrup this works in a Splunk clustered installation, but not in trial / single instance yet. It is a known issue that we are working on.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

netrc — Wed, 24 Feb 2016 20:02:49 GMT

port 8088 worked for me, but note the documentation for the "curl" example - it uses the "-k" option to allow "insecure" SSL connections.

I got nodeJS to work by setting the "rejectUnauhtorized: false" option to the https.request function.

Nevertheless, I've still seen Splunk return 200 (OK) to the https POST but still drop data.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

gblock_splunk — Wed, 24 Feb 2016 22:07:26 GMT

@netrc what do you mean it is dropping data?

In terms of the self-signed cert, it might work for node, but it doesn't work for several other stacks we tested namely .NET and Java. The main issue is the type of ECC cert we used is not compatible with several of these stacks.

It has still not been fixed, but it will be soon / our cloud team is working on it.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

jkat54 — Sat, 27 Feb 2016 21:15:54 GMT

As HEC is newer I have no experience with it... Just know 8088 is easy to typo as 8089 and vice versa.

Looks like you figured it out. Sorry but lots of issues with free Splunk and integrations.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

jkat54 — Sat, 27 Feb 2016 21:17:20 GMT

I agree. Sorry we couldn't make it work. Probably works fine with Splunk enterprise if you want to test. Also I covered your comment to the answer. Please mark it as such if you can.

Re: HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

ncrisler — Thu, 21 Apr 2016 15:42:28 GMT

Do you know if this has been fix or if there is a workaround?