topic Re: Events posted to HEC successfully, not showing up in index in Splunk Enterprise

Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 18:32:54 GMT

I am able to use the supplied official documentation on posting to a configured HEC, and I receive the correct success response, however the events are never indexed.

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 19:11:20 GMT

I am not seeing any warnings or errors anywhere. Just unable to find the test messages in any index at all anywhere after the successful response that the event was posted.

Re: Events posted to HEC successfully, not showing up in index

pradeepkumarg — Mon, 09 Jul 2018 19:15:04 GMT

Is your HEC instance a heavy forwarder or indexer? Is the data going to a new index? Is the index created?
Search for a larger time range?

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 19:18:35 GMT

This is on standalone enterprise test instance. indexes are created and searchable.

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Tue, 29 Sep 2020 20:22:01 GMT

I am even seeing it was indexed in the _introspection logs, but just cant search it
{"datetime":"07-09-2018 11:59:27.627 -0700","log_level":"INFO","component":"HttpEventCollector","data":{"token_name":"hectest","series":"http_event_collector_token","transport":"http","format":"json","total_bytes_received":50,"total_bytes_indexed":14,"num_of_requests":1,"num_of_events":1,"num_of_errors":0,"num_of_parser_errors":0,"num_of_requests_to_disabled_token":0,"num_of_requests_in_mint_format":0}}

Re: Events posted to HEC successfully, not showing up in index

pradeepkumarg — Mon, 09 Jul 2018 19:55:35 GMT

Did you expand your time range and search for a longer period to account for any timestamp/timezone issues?

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 19:58:23 GMT

Searching all time, still nothing

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 19:58:45 GMT

I should note, I am admin, full privilages

Re: Events posted to HEC successfully, not showing up in index

PowerPacked — Mon, 09 Jul 2018 20:06:32 GMT

Hi @Cuyose

check if the events are going to the default -- main index.

& u can see in - settings - data - indexes - your indexname --- & see if it recieved any events.

& also settings - data - data inputs - HTTP Event Collector - to see what was the index & token & many other settings.

Thanks

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 20:33:20 GMT

Thanks, all of this was checked, data is not showing up in any index. My best assumption is it may be permissions related, either having access to view or within the backend with permissions to write. As I stated _introspection shows the metrics of the event it saw come in successfully, but on the index management page within splunk, no events are being added to any index.

Is there some condition where an event will not get written to an index, and not log an error in _internal?

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Mon, 09 Jul 2018 21:24:27 GMT

Ok,

I got this to work. It would have been much easier with a one sentence edit in the documentation. The documented examples are in json, however when setting up the token you get to also select a sourcetype. I selected _json and it did not work. It was looking for a timestamp. You need to select _json_no_timestamp for the examples to work as documented.

Re: Events posted to HEC successfully, not showing up in index

andrewb_splunk — Mon, 09 Jul 2018 21:37:12 GMT

Cuyose, the Splunk doc team is very receptive to feedback. If you submit your suggestion using the feedback form on the bottom of the topic that needs improvement, the doc team will receive it and can take action on it. Sorry that the process was more difficult than it needed to be!

Re: Events posted to HEC successfully, not showing up in index

gjanders — Mon, 09 Jul 2018 23:01:42 GMT

however when setting up the token you
get to also select a sourcetype

Wouldn't the issue here be that your using a field extracted sourcetype that requires a time field? I believe a non-field extracted sourcetype would have worked just fine (or even not specifying the sourcetype at all).

However you could provide feedback to the docs team around this

Re: Events posted to HEC successfully, not showing up in index

Cuyose — Tue, 10 Jul 2018 14:35:32 GMT

That's what I thought, however when providing a time in the event Json, and selecting the _json structured sourcetype, it would not work, as none of the examples spoke to the importance of the selected sourcetype when following the provided examples, this wasted a lot of troubleshooting time.

I would suggest that the sample curl in the doc's include a time element, as that is key to how Splunk works, rather than simply sending a string message, which shows a successful post but does not get indexed.

Re: Events posted to HEC successfully, not showing up in index

woodcock — Tue, 10 Jul 2018 15:20:27 GMT

Please do click Accept on your answer and do include all of this feedback on the docs page.

Re: Events posted to HEC successfully, not showing up in index

gjanders — Tue, 10 Jul 2018 22:42:17 GMT

As per woodcock please accept your answer, however I've used that test curl command before without issue which implies that it might be something related to your chosen sourcetype.

Anyway, the docs team is open to feedback!

Re: Events posted to HEC successfully, not showing up in index

sujith549 — Thu, 07 Mar 2019 13:49:33 GMT

Hi @Cuyose

Do you got fixed the issue?, we are facing same issue verified all the Token,Index names. Able to see event got logged into _introspection.log but not able see the event info at any index( main or customIndex)

Re: Events posted to HEC successfully, not showing up in index

prasannjeet — Thu, 23 Jan 2020 14:09:24 GMT

Hey PowerPacked, thanks for your answer. I see that the events are logged successfully in the index. It has indeed received all the events that I sent. However, I don't know how to see those events. I am logged in as admin. Any ideas? Thanks!

Re: Events posted to HEC successfully, not showing up in index

prasannjeet — Thu, 23 Jan 2020 15:04:41 GMT

Nevermind, I found the problem. Firstly, it was because the index was not set to "main", and the other was the host url. It wasn't "https". Fixing both made the logs pop up in my dashboard. Thank you!