Splunk HEC Data Ingestion Issue from Azure Web Apps to Customer Cloud Instance

[coddydaddy88]

Environment

• Sending from: Azure Web Apps (Kudu CLI)
• Target: Two Splunk instances
  1. Personal trial Splunk (working)
  2. Customer's Splunk Cloud (not receiving data)

Problem Description

I am attempting to send log data to two different Splunk HTTP Event Collector (HEC) endpoints from an Azure Web Apps environment. I am using identical cURL commands for both:

Working Splunk Instance (Trial)

curl -k "https://[TRIAL_SPLUNK_HEC_URL]/services/collector/event" \
     -H "Authorization: Splunk [REDACTED_TOKEN]" \
     -d '{ "event": { ... }, "sourcetype": "_json", "index": "[TRIAL_INDEX]" }'

Non-Working Splunk Instance (Customer Cloud)

curl -k "https://[CUSTOMER_SPLUNK_HEC_URL]/services/collector/event" \
     -H "Authorization: Splunk [REDACTED_TOKEN]" \
     -d '{ "event": { ... }, "sourcetype": "_json", "index": "[CUSTOMER_INDEX]" }'

Observations

• Identical cURL command structure.
• For both Splunk instances, the cURL command on the CLI returns {"text":"Success","code":0}
• Despite the success response, data is successfully ingested into my personal trial Splunk instance, but no data appears in the customer's Splunk Cloud instance.

Questions

1. Given that the CLI returns "Success" for both, what could be preventing the data from appearing in the customer's Splunk Cloud instance?
2. What additional troubleshooting steps can verify HEC configuration and permissions on the customer's Splunk Cloud?
3. Could this issue be related to specific user role permissions or privileges, as I have been granted only limited access by the customer? Are there particular Splunk roles or permissions required for HEC token usage that might be missing?

Attempted Troubleshooting

• Verified correct HEC URL for both instances.
• Confirmed valid HEC tokens for both instances.
• Checked general network connectivity from the Azure Web Apps environment.
• Validated JSON payload structure.

Any guidance on identifying and resolving this selective data ingestion issue, especially considering the "Success" response from the CLI would be greatly appreciate

[coddydaddy88 · Accepted solution]

@livehybrid @PickleRick @kiran_panchavat Thank you so much for the responses and suggestions.  Albeit the issue was solved.

It turned out to be a rather funny situation! The data was actually being ingested into Splunk, as indicated by the 200 status code I was receiving for both the trial and customer Splunk instances. This aligns with what @PickleRick mentioned about the HEC endpoint returning 200 if it received the events properly. 

The problem was resolved when I tried double quotes when searching in Splunk (i.e., index="index_name" instead of index='index_name')🤦‍♂️ Once I did that, the data immediately became visible. I had the necessary access all along; it was a simple search syntax oversight.

I will also take note of @PickleRick 's advice regarding not using the _json sourcetype, as it's considered a bad practice. Thank you all again for your valuable insights!

[coddydaddy88]

@PickleRick @kiran_panchavat Thank you for you responses.

• @PickleRick I think you are correct I am trying to asses this on the customer's side "But there could be some routing/filtering happening on the receiving side."
  
• @kiran_panchavat In this article https://splunk.my.site.com/customer/s/article/Unable-to-ingest-data-though-the-HEC-token
  they suggested searching for index="_internal" level="ERROR" *timeout*, but I am getting zero results for that. This probably means I do not have access to that index

Another possible cause I am thinking is that , maybe the Customer side HEC might not have the specified index in the allowed indexes.

[livehybrid]

Hi @coddydaddy88 

You can test that your HEC token is working with the remote Splunk instance with the health endpoint, it should provide the following response:

{"text":"HEC is healthy","code":17}

 https://yoursplunkinstance/services/collector/health?token=31a1735d-2ba6-48bd-9098-cadcf9001f63

(Update token accordingly) 

This doesnt confirm the index is correct thought, as @PickleRick mentioned, it will return a 400 status if the index is not permitted. See https://docs.splunk.com/Documentation/Splunk/9.4.2/Data/TroubleshootHTTPEventCollector#:~:text=JSON%... for more info on status codes for HEC.

Can you do the following search, do you get any errors? 

index=_internal HttpInputDataHandler reply=*

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

As I wrote - if I remember correctly, if you post an event for an index you're not allowed you should get 400, not 200.

[PickleRick]

1. Do not use the default _json sourcetype. It has nothing to do with the problem at hand but it's a bad practice.

2. Generally, your reasoning is ok - as the HEC endpoint returns 200 that means that it received the event(s) properly and starts processing them. That doesn't mean anything beyond that.

I'm not sure but I suppose if you were sending data to an index your token does not have access to, you should be getting a 400 instead of 200 - see https://docs.splunk.com/Documentation/Splunk/9.4.2/Data/TroubleshootHTTPEventCollector#Possible_erro...

So your end seems ok.

But there could be some routing/filtering happening on the receiving side.

Or you might not have access to the index(es) as a user (Splunk will not throw any error at you; it will just say "no results").

Or you might be searching at wrong point in time (you seem to be sending events without explicit timestamp. The timestamp will not be parsed out from the event.)

[kiran_panchavat]

@coddydaddy88

• Confirm that [CUSTOMER_INDEX] exists.
• Ensure your user role has read access to that index.
• Please check internal logs

[kiran_panchavat]

@coddydaddy88 

Splunk Cloud may have an IP allow list configured, restricting HEC requests to specific IPs. If the Azure Web Apps outbound IP addresses are not included, requests may be accepted but not processed

https://splunk.my.site.com/customer/s/article/Unable-to-ingest-data-though-the-HEC-token