We are getting below errors on PCF end when sending data to Splunk HEC:
message":"splunk-nozzle-logger.Unable to talk to Splunk","log_level":2,"data":{"error":"Post http://xxxxxxxxxx:8088/services/collector: read tcp xxxxx:xxxx-\u003exxxxxx:xxxx: read: connection reset by peer"}}
message":"splunk-nozzle-logger.Encountered close error while reading from Firehose","log_level":2,"data":{"error":"websocket: close 1006 (abnormal closure): unexpected EOF"}}
I suspect both the errors are interlinked. When the client sends a message and crashes, then we get this EOF error. So the issue here is that the client is getting crashed after sending the message.
For more information go through below link:
https://github.com/gorilla/websocket/issues/321
Since the nozzle is running on PCF end, we won't be able to determine why it's crashing. Need to check with Pivotal to determine the root cause.
We get a similar error, but could not understand who is the client.
In our case, the application is in PCF, the PCF loggaggregator / firehose connects to the Splunk nozzle which sends the message to the Splunk enterprise.
We get the error in the splunk nozzle.