first Splunk install - cannot get HEC working

[ssdarkside2]

I set up a sample VM for myself to test out Splunk configuration. I wanted a stand-alone service just to make sure I can get my basic configuration running and forward logs from a Kubernetes instance. However, I am stuck in verification of the event receive resource.

Here's the steps I followed:

1. Setup a Linux VM
2. Get Splunk installed
3. Confirm web is working as expected
4. Create an index called splunk_test_events that is of (Type: events, App: search)
5. Go to Settings > Forwarding and Receiving and set up a port for 9997
6. In Settings > Data Inputs set up an HTTP Event Collector (details below)
7. Ensure tokens are enabled (I forget where this was)
8. Restart Splunk
9. SSH into the machine and check the running ports (see below)
10. Attempt to curl and event

So the HTTP Event Collector I set up as:

Name	splunk_testing_events
Source Type	Entered Source Type
Selected Allowed Indexes	splunk_test_events
Default Index	splunk_test_events
Output Group	None
Enable Indexer Acknowledgement	On

I verified that the HTTP Event Collector is enabled.

I log into the machine and check the ports that are active:

$ sudo lsof -i -P -n | grep LISTEN
systemd-r   649 systemd-resolve   13u  IPv4  23727      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd        751            root    3u  IPv4  26648      0t0  TCP *:22 (LISTEN)
sshd        751            root    4u  IPv6  26650      0t0  TCP *:22 (LISTEN)
splunkd    6405            root    4u  IPv4  63003      0t0  TCP *:8089 (LISTEN)
splunkd    6405            root   60u  IPv4  63818      0t0  TCP *:9997 (LISTEN)
splunkd    6405            root  128u  IPv4 123397      0t0  TCP *:8088 (LISTEN)
splunkd    6405            root  156u  IPv4  64895      0t0  TCP *:8000 (LISTEN)
mongod     6482            root   10u  IPv4  61364      0t0  TCP *:8191 (LISTEN)
python3.7  6623            root    7u  IPv4  63884      0t0  TCP 127.0.0.1:8065 (LISTEN)

Now I try and send a curl event over:

curl -v -k -H "Authorization: Splunk GENERATED_HEC_TOKEN" http://VM_PUBLIC_IP:9997/services/collector/event -d '{ "event": "testing manually" }' 

I get back an error:

*   Trying VM_PUBLIC_IP:9997...
* Connected to VM_PUBLIC_IP (VM_PUBLIC_IP) port 9997 (#0)
> POST /services/collector/event HTTP/1.1
> Host: VM_PUBLIC_IP:9997
> User-Agent: curl/7.74.0
> Accept: */*
> Authorization: Splunk GENERATED_HEC_TOKEN
> Content-Length: 31
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 31 out of 31 bytes
* Empty reply from server
* Connection #0 to host VM_PUBLIC_IP left intact
curl: (52) Empty reply from server

I tried some of the other ports:

• 8088: Connection reset by peer
• 8089: Connection reset by peer
• 8000: HTTP/1.1 303 (which I expected in this case)

What am I doing wrong here? 

[richgalloway]

The HEC port is 8088 by default so stick with that one unless you've explicitly changed it.

Check your firewalls to make sure port 8088 is reachable on that server.

[ssdarkside2]

Confirmed this again just to be sure:

$ curl -v -k -H "Authorization: Splunk GENERATED_TOKEN" http://localhost:8088/services/collector/event -d '{ "event": "testing manual upload" }'

*   Trying 127.0.0.1:8088...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8088 (#0)
> POST /services/collector/event HTTP/1.1
> Host: localhost:8088
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Splunk GENERATED_TOKEN
> Content-Length: 36
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 36 out of 36 bytes
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

[ssdarkside2]

I did try that, and I tried to post from localhost and got the same response.

[terry_berryhill]

Got same results as you, try https not http