Getting Data In

HEC troubleshooting in distributed Enterprise environment

cmuesing
Engager

I cannot figure out which component to enable HEC and where to send the events. We have an on prem Splunk Enterprise distributed configuration with a Deployment server, Indexer and SearchHead. We also have an independent "sandbox" environment for testing where I'm trying to set this up. Sandbox is 1 server with the whole Splunk Enterprise installation, however we do use the deployment server to setup and configure the sandbox universal forwarders, etc. 

I setup HEC tokens on the sandbox and could not figure out how to get it working. I am testing using Curl commands. I then added HEC tokens on the deployment server and still testing with Curl, cannot figure out how to send events to it. 

I get these errors:

1) Sending curl to sandbox URL with either deployment server HEC token or sandbox HEC token

"The requested URL was not found on this server.","code":404

2) Sending curl to indexer URL with either deployment server HEC token or Sandbox HEC token

Failed to connect to spidxa.open-techs.local port 8088: Connection refused

3) Sending curl to deployment server URL with either deployment server HEC token or Sandbox HEC token

Failed to connect to spmgta.open-techs.local port 8088: Connection timed out

4) Sending curl to SearchHead URL with either deployment server HEC token or Sandbox HEC token, and this is likely a firewall issue, but it doesn't make sense to me to send the event to the search head, so I haven't pushed security to open this port.

Failed to connect to spsha.open-techs.local port 8088: No route to host

This is my curl command with escaped double quotes and {variable substitutions}

curl -g -k --location --request POST 'https://#{server I am testing}:8088/services/collector/event' --header "Authorization: Splunk {token}" --header "Content-Type: text/plain" --data-raw "{\"event\": \"Test kong_dev\"}"

Can anybody tell me which components do which part of the HEC event collection? The introspection\http_event_Collector_metrics.log on both deployment and sandbox just show one minute intervals of 0 transactions going through there. 

 

Labels (1)
0 Karma
1 Solution

nortonjco
Explorer

Hi Cindy,

Setup the HEC on your sandbox and use the IP or hostname of the sandbox server and token values in the following command:

curl -k  https://<sandbox_IP_or_hostname_here>:8088/services/collector/event -H "Authorization: Splunk <your_token_value_from_HEC_input_here>" -d '{"event": "This is a test of HEC"}'

View solution in original post

0 Karma

nortonjco
Explorer

Hi Cindy,

Setup the HEC on your sandbox and use the IP or hostname of the sandbox server and token values in the following command:

curl -k  https://<sandbox_IP_or_hostname_here>:8088/services/collector/event -H "Authorization: Splunk <your_token_value_from_HEC_input_here>" -d '{"event": "This is a test of HEC"}'

0 Karma

cmuesing
Engager

Thank you so much nortonjco. That's just what I needed to get me past my issue. 

0 Karma

cmuesing
Engager

Can anybody tell at least which component  the HEC token needs to be created on and which component I send the event to? 

0 Karma

verbal_666
Contributor

There are many tutorials about this, and Splunk Documentation.

You need a Splunk Enterprise/Cloud (any instance, the instance, if not an Indexer, must obviously have an outputs.conf to the Indexer[s]).

You can create it in WebIf (Settings/Data inputs/HTTP Event Collector) with the wizard or in shell, inputs.conf must contain something like

[http://TestTOKEN]
index = your_choosen_default_index
indexes = your_wanted_available_indexes
token = your_token (something like f6823587-1222-4cf0-ad8a-324b6def6d8d)
disabled = 0

Events, then, can be posted as described in the thread with a POST with personal token Authentication.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...