Stuck on Splunk Cloud HEC Connection: DNS Resolution Issues

[coddydaddy88]

Hello Splunk Community,

I am trying to trace a Python FastAPI RAG API running on my local PC to my Splunk Cloud instance using the HTTP Event Collector (HEC). I am facing persistent issues with DNS resolution for the HEC endpoint hostname.

What I have done on Splunk Cloud:

1. Enabled HTTP Event Collector (HEC) in Global Settings.
2. Confirmed that SSL is required and enabled (cannot be unchecked).
3. Set the HTTP Port Number to 8088 in Global Settings.
4. Created an HEC token named "rag_tracer".
5. Assigned the token to use _json as the Sourcetype and mgap_cit_be_dev as the Index.

My local setup:

My application is a Python FastAPI service. I manage environment variables via a `.env` file, which is loaded at application startup. My tracing logic is implemented in a custom `tracing_helper.py` module, which attempts to connect to the HEC endpoint using the configured environment variables.

The Issue:

Despite configuring the HEC token and settings in Splunk Cloud, my application and direct `curl` commands consistently fail to connect to the HEC endpoint when using the hostname, resulting in `NameResolutionError` (or `[Errno 8] nodename nor servname provided, or not known`).

I have tried various `SPLUNK_HEC_URL` formats in my `.env` file, including:

• https://http-inputs-prd-p-gu5st.splunkcloud.com:8088/services/collector
• https://http-inputs-prd-p-gu5st.splunkcloud.com:443/services/collector
• https://inputs-prd-p-gu5st.splunkcloud.com:8088/services/collector
• https://prd-p-gu5st.splunkcloud.com:8088/services/collector
• https://prd-p-gu5st.splunkcloud.com:443/services/collector (This returned a 303 See Other, redirecting to `/en-US/services/collector`)

My application's logs repeatedly show:

DNS Resolution Error: [Errno 8] nodename nor servname provided, or not known
Unable to resolve Splunk Cloud HEC hostname. Check network configuration.
HTTPSConnectionPool(host='http-inputs-prd-p-gu5st.splunkcloud.com', port=443): Max retries exceeded with url: /services/collector (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x...>: Failed to resolve 'http-inputs-prd-p-gu5st.splunkcloud.com' ([Errno 8] nodename nor servname provided, or not known)"))

I have also performed network-level troubleshooting from my local machine:

• `nslookup http-inputs-prd-p-gu5st.splunkcloud.com` resulted in `** server can't find http-inputs-prd-p-gu5st.splunkcloud.com: REFUSED`
• `ping -c 4 http-inputs-prd-p-gu5st.splunkcloud.com` resulted in `ping: cannot resolve http-inputs-prd-p-gu5st.splunkcloud.com: Unknown host`
• `nslookup prd-p-gu5st.splunkcloud.com` (the main Splunk Cloud domain) also showed `REFUSED` for DNS resolution.
• However, `curl -v https://prd-p-gu5st.splunkcloud.com` successfully connected and resolved to `52.202.25.197`.
• Crucially, when I configured `SPLUNK_HEC_URL` in my `.env` file to use the direct IP address `https://52.202.25.197:8088/services/collector`, a `curl` command to HEC succeeded with `{"text":"Success","code":0}`. This confirms that the HEC endpoint is reachable via IP, but hostname resolution is the core problem.

My questions to the community are:

1. What is the exact and correct HEC endpoint URL format for Splunk Cloud trial instances (e.g., `prd-p-gu5st.splunkcloud.com`)? I cannot find a clear example of the full HEC URL in my Splunk Cloud console.
2. Can you outline the correct process to set up HEC in Splunk Cloud, step-by-step, specifically for a trial account?
3. What are the client-side configuration steps (e.g., environment variables, code changes) required to correctly communicate with HEC in Splunk Cloud from a local Python application?
4. Is there any specific IP allow-listing that needs to be done on the Splunk Cloud side for HEC, even though the documentation states it's open by default and I can access the Splunk Cloud web interface?

Any guidance or suggestions would be greatly appreciated.

Thank you!

[livehybrid]

In relation to your other questions:

• Can you outline the correct process to set up HEC in Splunk Cloud, step-by-step, specifically for a trial account?
  • The process you carried out as per your post is a valid/correct approach.
• What are the client-side configuration steps (e.g., environment variables, code changes) required to correctly communicate with HEC in Splunk Cloud from a local Python application?
  • The environment variables/code changes for your python application will ultimately depend on what you're implementing. If you'd like to share the code then we can help further, otherwise the only main difference between a Trial and Production stack is the endpoint (incl port 8088 for trial, but 443 for production) and also the SSL Verification which will need to be false for Trial stacks.
• Is there any specific IP allow-listing that needs to be done on the Splunk Cloud side for HEC, even though the documentation states it's open by default and I can access the Splunk Cloud web interface?
  • Splunk Cloud Trial stacks are open by default, Production environments are built with HEC open by default but can be limited if required using Admin Config Service (ACS) or settings within your cloud stack.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[livehybrid · Accepted solution]

Hi @coddydaddy88 

Splunk Cloud Trial instances are slightly different in configuration to production instances and have various restrictions.

Ultimately for a trial stack you need to use https://<stack>.splunkcloud.com:8088/services/collector/health

For your stack this would be: https://prd-p-gu5st.splunkcloud.com:8088/services/collector

We can confirm HEC is running and accessible by visiting: https://prd-p-gu5st.splunkcloud.com:8088/services/collector/health

Note however that there is no trusted SSL Cert on 8088 when accessing on a Splunk Trial stack, in production stacks this is done via a Load Balancer with a valid trusted public certificate.

For your trial instance you will need to accept/allow a non-trusted certificate by setting verify=false in your code.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

But the address http-inputs-prd-p-gu5st.splunkcloud.com doesn't resolve to an IP address indeed.

So it is not a proper address for the stack.

BTW, the trial stack uses a self-signed cert on HEC input or doesn't use TLS at all? I don't remember.

[livehybrid]

Thats right, the trial stacks run as a single instance without a load balancer infront of them I believe, therefore you access via the stack name on port 8088 without the http-input prefix. 

SSL is enabled on HEC but because it doesnt have a trusted cert (it uses the default Splunk cert on the server).