I have a heavy forwarder onprem installed on a windows OS.
I am troubleshooting why logs are not coming into the splunk cloud indexer from a cloud service over API. The api is between my onprem splunk heavy forwarder and the cloud service. I suspect the problem is on the cloud service side. I need a way to tell if the logs are even making it to my heavy forwarder. Is there a way to tail a running log on the heavy forwarder?
Also I am referring to the onprem slunk server as a heavy forwarder. Is that the proper term? It sends data to the cloud indexer.
Look in \Program Files\Splunk\var\log\splunk\splunkd.log for errors connecting to Splunk Cloud. They'll probably be associated with the TcpOutputProc component. If the HF has its web server enabled (it is by default) then you can sign in and search for
index=_internal (component=TcpOutputProc OR SSL).
Problems connecting to Splunk Cloud are usually on the on-prem side. Firewalls often block connections. Certificates may be missing or in the wrong location. The OS may not support the right version of SSL. The logs should offer suggestions about the cause in your case.
Rich. Other apps are able to get logs into this Forwarder via REST API and the logs are searchable on splunk cloud indexer. That tells me that the Forwarder is probably ok. I just need a way to show proof so that I can go back to the vendor's app side.
Proof will be confirming that the logs are not making it to the Forwarder.
I still recommend checking firewalls at both ends. Verify the app has the right URI.
If the HF has its web server enabled (it is by default) then you can sign in and search for
index=_internal component=TcpInputProc for errors or warnings about incoming connections.