After Splunk forwarder version got upgrade from 9.0.5.0 to 9.3.1.0 windows server are having issue in forwarding the data to Splunk.
Splunkd is stopping often in different server after restarting splund it start forwarding the data but issue comes again after 2,3 days
what actions to be taken to make the logs flow easily to Splunk
Firstly, check what happens - when the UF "stops", check what's at the end of splunkd.log to see whether anything out of the ordinary happened and see the windows system/application logs for entries regarding splunkd.exe to see if you see any indication of process crashing.
It might be a configuration issue but it indeed might be a software bug so you might end up calling support for help.
@PickleRick
error is something like Read error. An existing connection was forcibly closed by the remote host.
That might indicate issues with the receiving indexer. Check its logs and health.
@PickleRick
like while am searching in Splunk indexer am not able to see host, index and source for the windows server at that specific time.
No. I don't mean searching for the logs from the forwarder. This you won't find, it's obvious.
You need to look into _internal log for events from your receiving indexer(s) or HF(s) depending on what your infrastructure looks like concerning that disconnecting forwarder.
@PickleRick
so basically there is window server they had U.F which is forwarding data to Splunk.
While I checked the _internal logs not able to find anything on search head . What should I do next
There is no obvious answer. It might indeed require calling support.
The biggest change on the Universal forwarder from 9.0. > 9.3 was least-privileged user.
Do you see any issues on the permissions? I recommend working with support if this is happening frequently on all your windows hosts.
If this reply helps, Please UpVote.