I'm attempting to send a log event from a piece of Go code, but am receiving a connection refused error. If I use curl to send test data, I get an OK status back, but I don't see the data show up in my Splunk instance.
My request data looks like this (formatted pretty)
POST /services/collector HTTP/1.1
Host: input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088
Authorization: Splunk {token}
Content-Type: application/json
{"time":1450588381,"host":"Golang","source":"HTTPSplunkEvent","sourcetype":"Test","index":"tweet_harvest","event":"This is a test"}
My error comes back as:
--- FAIL: TestSendEvent (0.51s)
event_test.go:22: Error sending event: "Post https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector: remote error: handshake failure"
My code looks like this:
import (
"bytes"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/http/httputil"
"strconv"
)
//Event is a struct that holds all data to be sent to a Splunk HTTP logging
//endpoint
type Event struct {
Time int64 `json:"time"`
Host string `json:"host"`
Source string `json:"source"`
Sourcetype string `json:"sourcetype"`
Index string `json:"index"`
Event interface{} `json:"event"`
}
//Send the event to the specified Splunk Server
func (e Event) Send(destination string, token string, disableCertValidation bool) error {
//Ensure we have all of the values for the event
if e.Time == 0 || e.Host == "" || e.Source == "" || e.Sourcetype == "" || e.Index == "" || e.Event == nil {
return errors.New("All fields in Event must have a value")
}
//Create a byte array with the data
b, err := json.Marshal(e)
//Create client and request
client := &http.Client{}
request, err := http.NewRequest("POST", destination, bytes.NewBuffer(b))
if err != nil {
return err
}
//If we are working with a trial account the certificate will be self signed so we want to ignore certificate verification
if disableCertValidation {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client.Transport = tr
}
header := http.Header{}
header.Add("Authorization", "Splunk "+token)
header.Set("Content-Type", "application/json")
request.Header = header
resp, err := client.Do(request)
if err != nil {
dump, _ := httputil.DumpRequest(request, true)
fmt.Printf("Dumping request: %s\n", dump)
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
dump, err := httputil.DumpResponse(resp, true)
fmt.Printf("RESPONSE: %q\n", dump)
return err
}
//Any code other than 200 is an
var splunkResponse Response
decoder := json.NewDecoder(resp.Body)
err = decoder.Decode(&splunkResponse)
if err != nil {
fmt.Printf("Error decoding resposne\n")
return err
}
if splunkResponse.Code != SplunkResponseOK {
fmt.Printf("Splunk Response code not OK\n")
return errors.New(strconv.Itoa(splunkResponse.Code) + ": " + splunkResponse.Text)
}
//Everything is fine return nil error
return nil
}
And my test code:
import (
"testing"
"time"
)
func TestSendEvent(t *testing.T) {
event := &Event{
Time: time.Now().Unix(),
Index: "tweet_harvest",
Source: "HTTPSplunkEvent",
Sourcetype: "Test",
Event: "This is a test",
Host: "Golang",
}
err := event.Send("https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector",
"8CEB3C52-47B9-451E-81A9-4E45A299D41C", true)
if err != nil {
t.Fatalf("Error sending event: %#v\n", err.Error())
}
}
So I think that I've narrowed this down to Go doesn't support the self signed certificate used by the trial version of Splunk Cloud (ECDH-ECDSA-AES256-GCM-SHA384).
The list of supported cipher suites for go is:
TLS_RSA_WITH_RC4_128_SHA uint16 = 0x0005
TLS_RSA_WITH_3DES_EDE_CBC_SHA uint16 = 0x000a
TLS_RSA_WITH_AES_128_CBC_SHA uint16 = 0x002f
TLS_RSA_WITH_AES_256_CBC_SHA uint16 = 0x0035
TLS_ECDHE_ECDSA_WITH_RC4_128_SHA uint16 = 0xc007
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA uint16 = 0xc009
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA uint16 = 0xc00a
TLS_ECDHE_RSA_WITH_RC4_128_SHA uint16 = 0xc011
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA uint16 = 0xc012
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA uint16 = 0xc013
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA uint16 = 0xc014
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 uint16 = 0xc02f
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 uint16 = 0xc02b
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc030
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c
so I'm going to have to find an alternative way to do this. That said, I suspect that my code would work if I had a paid instance of Splunk with real certificates, which is more than I can commit to for an experimental project.
I fixed the problem with Curl by specifying the index. So I'm down to just the handshake problem with Go.
I see what you are saying about the port difference, and respect that you have way more experience, but I don't think the documentation is in error. If I change the port numbers when using curl I get the correct result with 8088 and an error with 8089.
curl -k https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8088/services/collector/event -H "Authorization: Splunk 8CEB3C52-47B9-451E-81A9-4E45A299D41C" -d '{"event": "hello world"}'
Returns:
{"text":"Success","code":0}
which is what you want.
While:
curl -k https://input-prd-p-zd5ktsgk9g47.cloud.splunk.com:8089/services/collector/event -H "Authorization: 8CEB3C52-47B9-451E-81A9-4E45A299D41C" -d '{"event": "hello world"}'
returns:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">Unauthorized</msg>
</messages>
</response>
I think the difference is that the event collector is using token based authentication rather than the REST client's user/password authentication.
I still have two problems. One, my code still gets a handshake failure with 8088. Two, the event I sent with Curl doesn't show up inside of my Splunk instance.
-Andy
As HEC is newer I have no experience with it... Just know 8088 is easy to typo as 8089 and vice versa.
Looks like you figured it out. Sorry but lots of issues with free Splunk and integrations.
The HEC will you 8088 by default. REST API is 8089. In this case, you are using the HEC (HTTP Event Collector) so 8088 is the correct port.
Did you setup the input with authentication from the Splunk Cloud instance?
Esix,
I created a HEC token as directed by the documentation, but Go doesn't appear to support the cipher suite that the Splunk Cloud Trial comes with, unless their is a way to configure the cipher suite myself in the cloud, I think I'm out of luck for use of HEC.
I think I can drop back and use the REST API instead, but it is going to take some rewriting.
-Andy
You're so close!