Splunk HEC Not Working

[Cerum]

So I'm unable to get HEC logs into Splunk Cloud (version 9.1.2312.102).

When I test the HECs in Postman via: (obviously didn't enter my domain or token for privacy reasons).

POST  https://http-inputs-mydomain.splunkcloud.com:443/services/collector/raw with the Authorization Header of "Splunk mytoken"

It works as expected and I receive a "text":Success , "code": 0 response, which is good.  I can also see the event in Splunk when I search it. 

I did this invidivdually for each HEC that I've created, and they all work....however, whenever I go to setup the actual HECs via the applications I'm trying to integrate...I get nothing.

I'm trying to send logs from Dashlane, FiveTran, Knowbe4, and OneTrust.  All of these support native Splunk integrations, I enter the information as requested on their external logging setup and nothing shows in Splunk.  I'm not sure what to do here.

Any guidance would be awesome!

Thanks in advance!

[deepakc]

Hi @Cerum 

You didn't mention IP allow lists checks, so might be worth checking your cloud  IP allow list config . In the past this has caught me out, for all your Apps (I'm assuming SaaS types) send to HEC cloud, therefore you may need to add them to your IP Allow list for the Splunk cloud feature (HEC access for ingestion), that is if you are even using IP allow lists, if you haven't then all the features are accessible and this is not the issue. 

https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Config/ConfigureIPAllowList 

[PickleRick]

Since you obviously can't do a tcpdump on the receiving side 😉 and I'm not sure about _internal contents in Cloud, you can either try to observe the traffic on the source side (as you're sending to the Cloud using TLS, you're not gonna see the payload of course but you'll at least be able to see the overall request-response cycle or lack thereof.

You can also install a temporary local instance, mirror the configuration and try to test it in unencrypted form to verify if your source systems handle the posting to HEC well.

Also I'm not sure if you don't have to enable sending from an allowed set of IPs to be able to receive traffic in Cloud in the first place (but I'm not a Cloud expert, don't quote me on that ;-))

[Cerum]

Come on, flip the pickle, Morty, you're not gonna regret it! Haha!

Thanks for the reply.  I have not tried actually capturing/sniffing traffic yet, although I'm headed in that direction.  As far as allowed IPs (for HEC ingestion) I set it to allow all for my testing, so I don't think that's the issue.

[PickleRick]

BTW, I just noticed, you're testing with the /raw endpoint. If the solutions you're trying to get events from claim to support "native Splunk HEC functionality",  they might be trying to post to the /event endpoint. And if they do it wrong, the input won't accept the data.

[Cerum]

I've tried both raw and event,  no joy.

[PickleRick]

OK. The easiest thing would indeed be to try to push to the /raw endpoint from your solution to verify whether anything is being sent at all (and checking any available logs on the sender's side if there are problems).

Aaaaand did you check for the usual culprit of "missing data" - time misconfiguration? It's a fairly common issue that the data is being indexed but it's just indexed at wrong moment in time so that you're not finding it properly. (it's more obvious if you index it ahead of time because then you can find it after some time if your source is constantly sending events but if it's indexing data "late", you won't find it if you're intuitively search for "last 30 minutes" or so).