We have configured a Splunk Enterprise distributed environment with the following components: 2 Search Heads (SH) 2 Indexers (IDX) 1 Management Node (acting as Cluster Manager, License Manager, and Deployer) 1 Heavy Forwarder (HF) 1 Deployment Server (DS) 1 SC4S instance During the SC4S configuration, we are seeing errors in Splunk searches indicating that logs are not being properly forwarded through the HTTP Event Collector (HEC). Below are some of the events we see in the sc4s:event sourcetype: - - syslog-ng 139 - [meta sequenceId="8133"] Server disconnected while preparing messages for sending, trying again; driver='d_hec_fmt#0', location='root generator dest_hec:5:5', worker_index='0', time_reopen='10', batch_size='84' host = splunk-sys01 source = sc4s sourcetype = sc4s:events - - syslog-ng 139 - [meta sequenceId="8131"] http: Server returned with a 4XX (client errors) status code, which means we are not authorized or the URL is not found; url='https://192.168.44.94:8088/services/collector/event', status_code='403', response='{"text":"Invalid token","code":4}', driver='d_hec_fmt#0', location='root generator dest_hec:5:5' host = splunk-sys01 source = sc4s sourcetype = sc4s:events We have already: Enabled HEC on the Splunk side Configured the HEC token Added the token and HEC endpoint in the SC4S env_file Restarted the SC4S service However, the logs indicate HTTP 403 – Invalid token errors when SC4S attempts to send events to the HEC endpoint. Question What could be causing the Invalid Token (HTTP 403) error in this setup, and what troubleshooting steps should we follow to resolve the issue so that SC4S can successfully forward logs to the indexers? "
... View more