HEC Events not indexing

[bsheppard8]

I'm learning how to use the HTTP Event collector, but no events ever show up in search. I have the inputs enabled and my token set up as shown:

When I run the command 'curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\"}"' in my command prompt, I get back {"text": "Success", "code": 0}.

I followed along with the tutorial on this site here: https://www.youtube.com/watch?v=qROXrFGqWAU

I've also tried changing the sourcetype to json_no_timestamp, but this didn't work either.

I'm confident that I've set up everything correctly, but nothing seems to be working. Is there a fix for this? Because I'm trying to do the same with collectd metrics.

[burwell]

Hi. As an experiment I used a deliberately bad token and found this error.  Try searching for errors.

index=_internal host=myindexers* log_level=ERROR component=HttpInputDataHandler

10-01-2021 23:00:15.133 +0000 ERROR HttpInputDataHandler - Failed processing http input, token name=n/a, channel=n/a, source_IP=1.2.3.4, reply=4, events_processed=0, http_input_body_size=39, parsing_err=""

[bsheppard8]

I tried using the search field by field. I was only able to find events with the index "_internal". Sadly, I wasn't able to find any events linked to events failing to process or issues with the HttpInputDataHandler. I don't see any issues in the splunkd log, either.

[PickleRick]

Is it some lab installation? Do you have high ingest ratio or rather "un-busy" system?

If it's a small installation, just do a realtime search for "index=*" and see whether (and where) your events appear. Don't try this on a busy server!

[bsheppard8]

I set it to an index that doesn't have any events so that I'd know right away that they're populating. And I have tried index=*, but I still don't see the test message I sent.

[PickleRick]

Check your /opt/splunk/var/log/splunkd.log for "HEC".

Typical error is that you send events to a non-existent index. But unless you have the destination index set to "Default" it's rather unlikely if you configure the input with GUI.

Anyway, add an "index" field to your HEC request and check if it works.

[bsheppard8]

The current index for the token is "history", and the default index is "main". I'm not seeing a log labelled "splunkd" for this instance, but are there configurations for the indices I could try?

[bsheppard8]

Update: I was able to find the log, but I'm not seeing anything about HEC so far.

[PickleRick]

But have you tried adding a "index" field to explicitly specify an index?

[bsheppard8]

Are you referring to the token? If so, yes, it's currently set to "history". I included a picture of the settings. If you mean something else, then no.

[PickleRick]

No. I mean instead of

curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\"}"

do

curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\",\"index\":\"history\"}"

[bsheppard8]

Yeah, I tried this too just now and searched for a matching index. Still nothing.

[PickleRick]

I noticed you're using /collector endpoint. Try /collector/event endpoint. I'm not sure - to be fully honest - what's the difference exactly, but there are two separate endpoints, so...

[bsheppard8]

I tried this as well, but I'm still not seeing any events. Is it possible that something in my instance isn't configured properly? Is there something I need to configure in order for an event to be created?

[PickleRick]

That's all interesting because... it should work but doesn't.

On HEC request you should either get an error (if you have bad token or try to write into an index you don't have permissions for) or the event should get accepted. You're saying that it does get accepted.

So it should either get written into an index or splunk itself should log something into logs that tells you what's preventing it from indexing the event (like trying to write to a non-existent index).