Getting Data In

HEC Events not indexing

bsheppard8
Loves-to-Learn Lots

I'm learning how to use the HTTP Event collector, but no events ever show up in search. I have the inputs enabled and my token set up as shown:

bsheppard8_0-1633091014316.png

When I run the command 'curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\"}"' in my command prompt, I get back {"text": "Success", "code": 0}.

I followed along with the tutorial on this site here: https://www.youtube.com/watch?v=qROXrFGqWAU

I've also tried changing the sourcetype to json_no_timestamp, but this didn't work either.

I'm confident that I've set up everything correctly, but nothing seems to be working. Is there a fix for this? Because I'm trying to do the same with collectd metrics.

Labels (2)
0 Karma

burwell
SplunkTrust
SplunkTrust

Hi. As an experiment I used a deliberately bad token and found this error.  Try searching for errors.

 

index=_internal host=myindexers* log_level=ERROR component=HttpInputDataHandler

10-01-2021 23:00:15.133 +0000 ERROR HttpInputDataHandler - Failed processing http input, token name=n/a, channel=n/a, source_IP=1.2.3.4, reply=4, events_processed=0, http_input_body_size=39, parsing_err=""

 

 

0 Karma

bsheppard8
Loves-to-Learn Lots
I tried using the search field by field. I was only able to find events with the index "_internal". Sadly, I wasn't able to find any events linked to events failing to process or issues with the HttpInputDataHandler. I don't see any issues in the splunkd log, either.
0 Karma

PickleRick
Champion

Is it some lab installation? Do you have high ingest ratio or rather "un-busy" system?

If it's a small installation, just do a realtime search for "index=*" and see whether (and where) your events appear. Don't try this on a busy server!

0 Karma

bsheppard8
Loves-to-Learn Lots
I set it to an index that doesn't have any events so that I'd know right away that they're populating. And I have tried index=*, but I still don't see the test message I sent.
0 Karma

PickleRick
Champion

Check your /opt/splunk/var/log/splunkd.log for "HEC".

Typical error is that you send events to a non-existent index. But unless you have the destination index set to "Default" it's rather unlikely if you configure the input with GUI.

Anyway, add an "index" field to your HEC request and check if it works.

0 Karma

bsheppard8
Loves-to-Learn Lots
The current index for the token is "history", and the default index is "main". I'm not seeing a log labelled "splunkd" for this instance, but are there configurations for the indices I could try?
0 Karma

bsheppard8
Loves-to-Learn Lots
Update: I was able to find the log, but I'm not seeing anything about HEC so far.
0 Karma

PickleRick
Champion

But have you tried adding a "index" field to explicitly specify an index?

0 Karma

bsheppard8
Loves-to-Learn Lots
Are you referring to the token? If so, yes, it's currently set to "history". I included a picture of the settings. If you mean something else, then no.
0 Karma

PickleRick
Champion

No. I mean instead of

curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\"}"

do

curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\",\"index\":\"history\"}"

 

0 Karma

bsheppard8
Loves-to-Learn Lots
Yeah, I tried this too just now and searched for a matching index. Still nothing.
0 Karma

PickleRick
Champion

I noticed you're using /collector endpoint. Try /collector/event endpoint. I'm not sure - to be fully honest - what's the difference exactly, but there are two separate endpoints, so...

0 Karma

bsheppard8
Loves-to-Learn Lots
I tried this as well, but I'm still not seeing any events. Is it possible that something in my instance isn't configured properly? Is there something I need to configure in order for an event to be created?
0 Karma

PickleRick
Champion

That's all interesting because... it should work but doesn't.

On HEC request you should either get an error (if you have bad token or try to write into an index you don't have permissions for) or the event should get accepted. You're saying that it does get accepted.

So it should either get written into an index or splunk itself should log something into logs that tells you what's preventing it from indexing the event (like trying to write to a non-existent index).

0 Karma