Monitoring Splunk

Not getting splunk logs

Praz_123
Communicator

After  Splunk forwarder version got upgrade from 9.0.5.0 to 9.3.1.0 windows server are having issue in forwarding the data to Splunk.

Splunkd is stopping often in different server after restarting splund it start forwarding the data but issue comes again after 2,3 days 

what actions to be taken to make the logs flow easily to Splunk

 

Labels (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Firstly, check what happens - when the UF "stops", check what's at the end of splunkd.log to see whether anything out of the ordinary happened and see the windows system/application logs for entries regarding splunkd.exe to see if you see any indication of process crashing.

It might be a configuration issue but it indeed might be a software bug so you might end up calling support for help.

0 Karma

Praz_123
Communicator

@PickleRick 

error is something like Read error. An existing connection was forcibly closed by the remote host.

Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

That might indicate issues with the receiving indexer. Check its logs and health.

0 Karma

Praz_123
Communicator

@PickleRick 

 like while am searching in Splunk indexer am not able to see host, index and source for the windows server at that specific time.

Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

No. I don't mean searching for the logs from the forwarder. This you won't find, it's obvious.

You need to look into _internal log for events from your receiving indexer(s) or HF(s) depending on what your infrastructure looks like concerning that disconnecting forwarder.

0 Karma

Praz_123
Communicator

@PickleRick 

so basically there is window server they had U.F which is forwarding data to Splunk.

While I checked the _internal logs not able to find anything on search head . What should I do next 

Tags (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

There is no obvious answer. It might indeed require calling support.

0 Karma

sainag_splunk
Splunk Employee
Splunk Employee

The biggest change on the Universal forwarder from 9.0. > 9.3 was least-privileged user.

https://docs.splunk.com/Documentation/Forwarder/9.3.1/Forwarder/InstallaWindowsuniversalforwarderfro...

 

Do you see any issues on the permissions? I recommend working with support if this is happening frequently on all your windows hosts.




If this reply helps, Please UpVote.

If this helps, Upvote!!!!
Together we make the Splunk Community stronger 
0 Karma
Get Updates on the Splunk Community!

Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud ...

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it ...

Real-Time Fraud Detection: How Splunk Dashboards Protect Financial Institutions

Financial fraud isn't slowing down. If anything, it's getting more sophisticated. Account takeovers, credit ...

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...