Getting Data In

first Splunk install - cannot get HEC working

ssdarkside2
Explorer

I set up a sample VM for myself to test out Splunk configuration. I wanted a stand-alone service just to make sure I can get my basic configuration running and forward logs from a Kubernetes instance. However, I am stuck in verification of the event receive resource.

Here's the steps I followed:

  1. Setup a Linux VM
  2. Get Splunk installed
  3. Confirm web is working as expected
  4. Create an index called splunk_test_events that is of (Type: events, App: search)
  5. Go to Settings > Forwarding and Receiving and set up a port for 9997
  6. In Settings > Data Inputs set up an HTTP Event Collector (details below)
  7. Ensure tokens are enabled (I forget where this was)
  8. Restart Splunk
  9. SSH into the machine and check the running ports (see below)
  10. Attempt to curl and event

So the HTTP Event Collector I set up as:

Namesplunk_testing_events
Source TypeEntered Source Type
Selected Allowed Indexessplunk_test_events
Default Indexsplunk_test_events
Output GroupNone
Enable Indexer AcknowledgementOn 

 

I verified that the HTTP Event Collector is enabled.

I log into the machine and check the ports that are active:

$ sudo lsof -i -P -n | grep LISTEN
systemd-r   649 systemd-resolve   13u  IPv4  23727      0t0  TCP 127.0.0.53:53 (LISTEN)
sshd        751            root    3u  IPv4  26648      0t0  TCP *:22 (LISTEN)
sshd        751            root    4u  IPv6  26650      0t0  TCP *:22 (LISTEN)
splunkd    6405            root    4u  IPv4  63003      0t0  TCP *:8089 (LISTEN)
splunkd    6405            root   60u  IPv4  63818      0t0  TCP *:9997 (LISTEN)
splunkd    6405            root  128u  IPv4 123397      0t0  TCP *:8088 (LISTEN)
splunkd    6405            root  156u  IPv4  64895      0t0  TCP *:8000 (LISTEN)
mongod     6482            root   10u  IPv4  61364      0t0  TCP *:8191 (LISTEN)
python3.7  6623            root    7u  IPv4  63884      0t0  TCP 127.0.0.1:8065 (LISTEN)

 

Now I try and send a curl event over:

curl -v -k -H "Authorization: Splunk GENERATED_HEC_TOKEN" http://VM_PUBLIC_IP:9997/services/collector/event -d '{ "event": "testing manually" }' 

 

I get back an error:

*   Trying VM_PUBLIC_IP:9997...
* Connected to VM_PUBLIC_IP (VM_PUBLIC_IP) port 9997 (#0)
> POST /services/collector/event HTTP/1.1
> Host: VM_PUBLIC_IP:9997
> User-Agent: curl/7.74.0
> Accept: */*
> Authorization: Splunk GENERATED_HEC_TOKEN
> Content-Length: 31
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 31 out of 31 bytes
* Empty reply from server
* Connection #0 to host VM_PUBLIC_IP left intact
curl: (52) Empty reply from server

 

I tried some of the other ports:

  • 8088: Connection reset by peer
  • 8089: Connection reset by peer
  • 8000: HTTP/1.1 303 (which I expected in this case)

What am I doing wrong here? 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The HEC port is 8088 by default so stick with that one unless you've explicitly changed it.

Check your firewalls to make sure port 8088 is reachable on that server.

---
If this reply helps you, Karma would be appreciated.
0 Karma

ssdarkside2
Explorer

Confirmed this again just to be sure:

 

$ curl -v -k -H "Authorization: Splunk GENERATED_TOKEN" http://localhost:8088/services/collector/event -d '{ "event": "testing manual upload" }'

*   Trying 127.0.0.1:8088...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8088 (#0)
> POST /services/collector/event HTTP/1.1
> Host: localhost:8088
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Splunk GENERATED_TOKEN
> Content-Length: 36
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 36 out of 36 bytes
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
0 Karma

ssdarkside2
Explorer

I did try that, and I tried to post from localhost and got the same response.

0 Karma

terry_berryhill
Loves-to-Learn Lots

Got same results as you, try https not http

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...