topic HEC Events not indexing in Getting Data In

HEC Events not indexing

bsheppard8 — Fri, 01 Oct 2021 12:31:58 GMT

I'm learning how to use the HTTP Event collector, but no events ever show up in search. I have the inputs enabled and my token set up as shown:

When I run the command 'curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\"}"' in my command prompt, I get back {"text": "Success", "code": 0}.

I followed along with the tutorial on this site here: https://www.youtube.com/watch?v=qROXrFGqWAU

I've also tried changing the sourcetype to json_no_timestamp, but this didn't work either.

I'm confident that I've set up everything correctly, but nothing seems to be working. Is there a fix for this? Because I'm trying to do the same with collectd metrics.

Re: HEC Events not indexing

PickleRick — Fri, 01 Oct 2021 13:02:29 GMT

Is it some lab installation? Do you have high ingest ratio or rather "un-busy" system?

If it's a small installation, just do a realtime search for "index=*" and see whether (and where) your events appear. Don't try this on a busy server!

Re: HEC Events not indexing

bsheppard8 — Fri, 01 Oct 2021 13:18:11 GMT

I set it to an index that doesn't have any events so that I'd know right away that they're populating. And I have tried index=*, but I still don't see the test message I sent.

Re: HEC Events not indexing

PickleRick — Fri, 01 Oct 2021 14:09:41 GMT

Check your /opt/splunk/var/log/splunkd.log for "HEC".

Typical error is that you send events to a non-existent index. But unless you have the destination index set to "Default" it's rather unlikely if you configure the input with GUI.

Anyway, add an "index" field to your HEC request and check if it works.

Re: HEC Events not indexing

bsheppard8 — Fri, 01 Oct 2021 15:30:24 GMT

The current index for the token is "history", and the default index is "main". I'm not seeing a log labelled "splunkd" for this instance, but are there configurations for the indices I could try?

Re: HEC Events not indexing

bsheppard8 — Fri, 01 Oct 2021 17:25:08 GMT

Update: I was able to find the log, but I'm not seeing anything about HEC so far.

Re: HEC Events not indexing

PickleRick — Fri, 01 Oct 2021 18:11:22 GMT

But have you tried adding a "index" field to explicitly specify an index?

Re: HEC Events not indexing

bsheppard8 — Fri, 01 Oct 2021 18:26:29 GMT

Are you referring to the token? If so, yes, it's currently set to "history". I included a picture of the settings. If you mean something else, then no.

Re: HEC Events not indexing

PickleRick — Fri, 01 Oct 2021 18:41:08 GMT

No. I mean instead of

curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\"}"

curl -k http://<instance-host>:8088/services/collector -H "Authorization:Splunk 4f99809e-55d3-4677-b418-c0be66693311" -d "{\"sourcetype\": \"trial\", \"event\":\"Hello World!\",\"index\":\"history\"}"

Re: HEC Events not indexing

bsheppard8 — Fri, 01 Oct 2021 18:46:26 GMT

Yeah, I tried this too just now and searched for a matching index. Still nothing.

Re: HEC Events not indexing

burwell — Fri, 01 Oct 2021 23:09:52 GMT

Hi. As an experiment I used a deliberately bad token and found this error. Try searching for errors.

index=_internal host=myindexers* log_level=ERROR component=HttpInputDataHandler 10-01-2021 23:00:15.133 +0000 ERROR HttpInputDataHandler - Failed processing http input, token name=n/a, channel=n/a, source_IP=1.2.3.4, reply=4, events_processed=0, http_input_body_size=39, parsing_err=""

Re: HEC Events not indexing

PickleRick — Sat, 02 Oct 2021 06:25:13 GMT

I noticed you're using /collector endpoint. Try /collector/event endpoint. I'm not sure - to be fully honest - what's the difference exactly, but there are two separate endpoints, so...

Re: HEC Events not indexing

bsheppard8 — Mon, 04 Oct 2021 11:27:24 GMT

I tried this as well, but I'm still not seeing any events. Is it possible that something in my instance isn't configured properly? Is there something I need to configure in order for an event to be created?

Re: HEC Events not indexing

bsheppard8 — Mon, 04 Oct 2021 11:31:06 GMT

I tried using the search field by field. I was only able to find events with the index "_internal". Sadly, I wasn't able to find any events linked to events failing to process or issues with the HttpInputDataHandler. I don't see any issues in the splunkd log, either.

Re: HEC Events not indexing

PickleRick — Wed, 06 Oct 2021 14:32:11 GMT

That's all interesting because... it should work but doesn't.

On HEC request you should either get an error (if you have bad token or try to write into an index you don't have permissions for) or the event should get accepted. You're saying that it does get accepted.

So it should either get written into an index or splunk itself should log something into logs that tells you what's preventing it from indexing the event (like trying to write to a non-existent index).