HEC troubleshooting in distributed Enterprise environment

[cmuesing]

I cannot figure out which component to enable HEC and where to send the events. We have an on prem Splunk Enterprise distributed configuration with a Deployment server, Indexer and SearchHead. We also have an independent "sandbox" environment for testing where I'm trying to set this up. Sandbox is 1 server with the whole Splunk Enterprise installation, however we do use the deployment server to setup and configure the sandbox universal forwarders, etc. 

I setup HEC tokens on the sandbox and could not figure out how to get it working. I am testing using Curl commands. I then added HEC tokens on the deployment server and still testing with Curl, cannot figure out how to send events to it. 

I get these errors:

1) Sending curl to sandbox URL with either deployment server HEC token or sandbox HEC token

"The requested URL was not found on this server.","code":404

2) Sending curl to indexer URL with either deployment server HEC token or Sandbox HEC token

Failed to connect to spidxa.open-techs.local port 8088: Connection refused

3) Sending curl to deployment server URL with either deployment server HEC token or Sandbox HEC token

Failed to connect to spmgta.open-techs.local port 8088: Connection timed out

4) Sending curl to SearchHead URL with either deployment server HEC token or Sandbox HEC token, and this is likely a firewall issue, but it doesn't make sense to me to send the event to the search head, so I haven't pushed security to open this port.

Failed to connect to spsha.open-techs.local port 8088: No route to host

This is my curl command with escaped double quotes and {variable substitutions}

curl -g -k --location --request POST 'https://#{server I am testing}:8088/services/collector/event' --header "Authorization: Splunk {token}" --header "Content-Type: text/plain" --data-raw "{\"event\": \"Test kong_dev\"}"

Can anybody tell me which components do which part of the HEC event collection? The introspection\http_event_Collector_metrics.log on both deployment and sandbox just show one minute intervals of 0 transactions going through there. 

[nortonjco · Accepted solution]

Hi Cindy,

Setup the HEC on your sandbox and use the IP or hostname of the sandbox server and token values in the following command:

curl -k  https://<sandbox_IP_or_hostname_here>:8088/services/collector/event -H "Authorization: Splunk <your_token_value_from_HEC_input_here>" -d '{"event": "This is a test of HEC"}'

[cmuesing]

Thank you so much nortonjco. That's just what I needed to get me past my issue. 

[cmuesing]

Can anybody tell at least which component  the HEC token needs to be created on and which component I send the event to? 

[verbal_666]

There are many tutorials about this, and Splunk Documentation.

You need a Splunk Enterprise/Cloud (any instance, the instance, if not an Indexer, must obviously have an outputs.conf to the Indexer[s]).

You can create it in WebIf (Settings/Data inputs/HTTP Event Collector) with the wizard or in shell, inputs.conf must contain something like

[http://TestTOKEN]
index = your_choosen_default_index
indexes = your_wanted_available_indexes
token = your_token (something like f6823587-1222-4cf0-ad8a-324b6def6d8d)
disabled = 0

Events, then, can be posted as described in the thread with a POST with personal token Authentication.