Getting Data In

HTTP Event Collector: How to troubleshoot why I'm getting a connection refused error?

anortrup
Explorer

I'm attempting to send a log event from a piece of Go code, but am receiving a connection refused error. If I use curl to send test data, I get an OK status back, but I don't see the data show up in my Splunk instance.

My request data looks like this (formatted pretty)

POST /services/collector HTTP/1.1
Host: input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088
Authorization: Splunk {token}
Content-Type: application/json
{"time":1450588381,"host":"Golang","source":"HTTPSplunkEvent","sourcetype":"Test","index":"tweet_harvest","event":"This is a test"}

My error comes back as:

--- FAIL: TestSendEvent (0.51s)
event_test.go:22: Error sending event: "Post https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector: remote error: handshake failure"

My code looks like this:

import (
    "bytes"
    "crypto/tls"
    "encoding/json"
    "errors"
    "fmt"
    "net/http"
    "net/http/httputil"
    "strconv"
)

//Event is a struct that holds all data to be sent to a Splunk HTTP logging
//endpoint
type Event struct {
    Time       int64       `json:"time"`
    Host       string      `json:"host"`
    Source     string      `json:"source"`
    Sourcetype string      `json:"sourcetype"`
    Index      string      `json:"index"`
    Event      interface{} `json:"event"`
}

//Send the event to the specified Splunk Server
func (e Event) Send(destination string, token string, disableCertValidation bool) error {

    //Ensure we have all of the values for the event
    if e.Time == 0 || e.Host == "" || e.Source == "" || e.Sourcetype == "" || e.Index == "" || e.Event == nil {
        return errors.New("All fields in Event must have a value")
    }

    //Create a byte array with the data
    b, err := json.Marshal(e)

    //Create client and request
    client := &http.Client{}
    request, err := http.NewRequest("POST", destination, bytes.NewBuffer(b))
    if err != nil {
        return err
    }

    //If we are working with a trial account the certificate will be self signed so we want to ignore certificate verification
    if disableCertValidation {
        tr := &http.Transport{
            TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
        }
        client.Transport = tr
    }

    header := http.Header{}
    header.Add("Authorization", "Splunk "+token)
    header.Set("Content-Type", "application/json")
    request.Header = header

    resp, err := client.Do(request)
    if err != nil {
        dump, _ := httputil.DumpRequest(request, true)
        fmt.Printf("Dumping request: %s\n", dump)
        return err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        dump, err := httputil.DumpResponse(resp, true)
        fmt.Printf("RESPONSE: %q\n", dump)
        return err
    }

    //Any code other than 200 is an
    var splunkResponse Response
    decoder := json.NewDecoder(resp.Body)
    err = decoder.Decode(&splunkResponse)

    if err != nil {
        fmt.Printf("Error decoding resposne\n")
        return err
    }

    if splunkResponse.Code != SplunkResponseOK {

        fmt.Printf("Splunk Response code not OK\n")
        return errors.New(strconv.Itoa(splunkResponse.Code) + ": " + splunkResponse.Text)
    }

    //Everything is fine return nil error
    return nil

}

And my test code:

import (
    "testing"
    "time"
)

func TestSendEvent(t *testing.T) {
    event := &Event{
        Time:       time.Now().Unix(),
        Index:      "tweet_harvest",
        Source:     "HTTPSplunkEvent",
        Sourcetype: "Test",
        Event:      "This is a test",
        Host:       "Golang",
    }

    err := event.Send("https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector",
        "8CEB3C52-47B9-451E-81A9-4E45A299D41C", true)

    if err != nil {
        t.Fatalf("Error sending event: %#v\n", err.Error())
    }
}
1 Solution

anortrup
Explorer

So I think that I've narrowed this down to Go doesn't support the self signed certificate used by the trial version of Splunk Cloud (ECDH-ECDSA-AES256-GCM-SHA384).

The list of supported cipher suites for go is:

    TLS_RSA_WITH_RC4_128_SHA                uint16 = 0x0005
    TLS_RSA_WITH_3DES_EDE_CBC_SHA           uint16 = 0x000a
    TLS_RSA_WITH_AES_128_CBC_SHA            uint16 = 0x002f
    TLS_RSA_WITH_AES_256_CBC_SHA            uint16 = 0x0035
    TLS_ECDHE_ECDSA_WITH_RC4_128_SHA        uint16 = 0xc007
    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA    uint16 = 0xc009
    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA    uint16 = 0xc00a
    TLS_ECDHE_RSA_WITH_RC4_128_SHA          uint16 = 0xc011
    TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA     uint16 = 0xc012
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA      uint16 = 0xc013
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA      uint16 = 0xc014
    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256   uint16 = 0xc02f
    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 uint16 = 0xc02b
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   uint16 = 0xc030
    TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c

so I'm going to have to find an alternative way to do this. That said, I suspect that my code would work if I had a paid instance of Splunk with real certificates, which is more than I can commit to for an experimental project.

View solution in original post

anortrup
Explorer

I fixed the problem with Curl by specifying the index. So I'm down to just the handshake problem with Go.

0 Karma

anortrup
Explorer

I see what you are saying about the port difference, and respect that you have way more experience, but I don't think the documentation is in error. If I change the port numbers when using curl I get the correct result with 8088 and an error with 8089.

curl -k https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector/event -H "Authorization: Splunk 8CEB3C52-47B9-451E-81A9-4E45A299D41C" -d '{"event": "hello world"}'

Returns:

{"text":"Success","code":0}

which is what you want.

While:

curl -k https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8089/services/collector/event -H "Authorization: 8CEB3C52-47B9-451E-81A9-4E45A299D41C" -d '{"event": "hello world"}'

returns:

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="ERROR">Unauthorized</msg>
  </messages>
</response>

I think the difference is that the event collector is using token based authentication rather than the REST client's user/password authentication.

I still have two problems. One, my code still gets a handshake failure with 8088. Two, the event I sent with Curl doesn't show up inside of my Splunk instance.

-Andy

jkat54
SplunkTrust
SplunkTrust

As HEC is newer I have no experience with it... Just know 8088 is easy to typo as 8089 and vice versa.

Looks like you figured it out. Sorry but lots of issues with free Splunk and integrations.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

The HEC will you 8088 by default. REST API is 8089. In this case, you are using the HEC (HTTP Event Collector) so 8088 is the correct port.

Did you setup the input with authentication from the Splunk Cloud instance?

anortrup
Explorer

Esix,

I created a HEC token as directed by the documentation, but Go doesn't appear to support the cipher suite that the Splunk Cloud Trial comes with, unless their is a way to configure the cipher suite myself in the cloud, I think I'm out of luck for use of HEC.

I think I can drop back and use the REST API instead, but it is going to take some rewriting.

-Andy

0 Karma

jkat54
SplunkTrust
SplunkTrust

You're so close!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...