HTTP Event Collector for SC4S is giving Error Message

[mosaddek]

We have configured a Splunk Enterprise distributed environment with the following components:

• 2 Search Heads (SH)

• 2 Indexers (IDX)

• 1 Management Node (acting as Cluster Manager, License Manager, and Deployer)

• 1 Heavy Forwarder (HF)

• 1 Deployment Server (DS)

• 1 SC4S instance

During the SC4S configuration, we are seeing errors in Splunk searches indicating that logs are not being properly forwarded through the HTTP Event Collector (HEC).

Below are some of the events we see in the sc4s:event sourcetype:

- - syslog-ng 139 - [meta sequenceId="8133"]
Server disconnected while preparing messages for sending, trying again;
driver='d_hec_fmt#0', location='root generator dest_hec:5:5',
worker_index='0', time_reopen='10', batch_size='84'

host = splunk-sys01
source = sc4s
sourcetype = sc4s:events

- - syslog-ng 139 - [meta sequenceId="8131"]
http: Server returned with a 4XX (client errors) status code, which means we are not authorized or the URL is not found;
url='https://192.168.44.94:8088/services/collector/event',
status_code='403',
response='{"text":"Invalid token","code":4}',
driver='d_hec_fmt#0', location='root generator dest_hec:5:5'

host = splunk-sys01
source = sc4s
sourcetype = sc4s:events

We have already:

• Enabled HEC on the Splunk side

• Configured the HEC token

• Added the token and HEC endpoint in the SC4S env_file Restarted the SC4S service

However, the logs indicate HTTP 403 – Invalid token errors when SC4S attempts to send events to the HEC endpoint.

Question

What could be causing the Invalid Token (HTTP 403) error in this setup, and what troubleshooting steps should we follow to resolve the issue so that SC4S can successfully forward logs to the indexers?

"

[livehybrid]

Hi @mosaddek 

Can you confirm the variable name used in SC4S env_file for the token please?

Also can you confirm you're using the GUID for the HEC token and not the name of it? The token should be something like d79f596e-2b07-46dc-a7e1-320d1e086580

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[mosaddek]

Solved the Problem by creating the HEC in CM, and push it to all the Indexers.

[PickleRick]

1. Why do you have a deployer if you have just two SHs (so no SH cluster).

2. Have you checked manually if you can post an event to your HEC endpoint? https://help.splunk.com/en/splunk-enterprise/get-data-in/collect-http-event-data/http-event-collecto...