topic Re: Not getting splunk logs in Monitoring Splunk

Not getting splunk logs

Praz_123 — Wed, 06 Nov 2024 16:54:38 GMT

After Splunk forwarder version got upgrade from 9.0.5.0 to 9.3.1.0 windows server are having issue in forwarding the data to Splunk.

Splunkd is stopping often in different server after restarting splund it start forwarding the data but issue comes again after 2,3 days

what actions to be taken to make the logs flow easily to Splunk

Re: Not getting splunk logs

sainag_splunk — Thu, 07 Nov 2024 04:56:17 GMT

The biggest change on the Universal forwarder from 9.0. > 9.3 was least-privileged user.

https://docs.splunk.com/Documentation/Forwarder/9.3.1/Forwarder/InstallaWindowsuniversalforwarderfromaninstaller#Manage_SePrivilegeUser_permissions

Do you see any issues on the permissions? I recommend working with support if this is happening frequently on all your windows hosts.

If this reply helps, Please UpVote.

Re: Not getting splunk logs

PickleRick — Thu, 07 Nov 2024 08:27:06 GMT

Firstly, check what happens - when the UF "stops", check what's at the end of splunkd.log to see whether anything out of the ordinary happened and see the windows system/application logs for entries regarding splunkd.exe to see if you see any indication of process crashing.

It might be a configuration issue but it indeed might be a software bug so you might end up calling support for help.

Re: Not getting splunk logs

Praz_123 — Thu, 07 Nov 2024 09:09:35 GMT

@PickleRick

error is something like Read error. An existing connection was forcibly closed by the remote host.

Re: Not getting splunk logs

PickleRick — Thu, 07 Nov 2024 09:12:36 GMT

That might indicate issues with the receiving indexer. Check its logs and health.

Re: Not getting splunk logs

Praz_123 — Thu, 07 Nov 2024 09:54:18 GMT

@PickleRick

like while am searching in Splunk indexer am not able to see host, index and source for the windows server at that specific time.

Re: Not getting splunk logs

PickleRick — Thu, 07 Nov 2024 10:36:30 GMT

No. I don't mean searching for the logs from the forwarder. This you won't find, it's obvious.

You need to look into _internal log for events from your receiving indexer(s) or HF(s) depending on what your infrastructure looks like concerning that disconnecting forwarder.

Re: Not getting splunk logs

Praz_123 — Fri, 08 Nov 2024 09:30:06 GMT

@PickleRick

so basically there is window server they had U.F which is forwarding data to Splunk.

While I checked the _internal logs not able to find anything on search head . What should I do next

Re: Not getting splunk logs

PickleRick — Fri, 08 Nov 2024 15:47:49 GMT

There is no obvious answer. It might indeed require calling support.