Forwarder Phoning Home but No Logs Found

[drizzo]

I have all fifty-two forwarders phoning home to the server. However, I can't see any logs from the hosts besides the server itself. So, the computer where the Enterprise is installed has no issues with its logs being captured, but every other host has the issue. I don't believe it's a Firewall complication since I've turned it off.

What I am looking for is some areas to trouble shoot this problem -- any pointers on where to look. I have three apps (_server_app_windowshost, splunk_TA_windows, and uf_outputs) which say they have all been deployed to 100% of the hosts.

I did copy over my uf_outputs and _server_app_windowshost files from my other ntowrk, edditted the IP address where needed, then placed them on the new server under deployment apps.

Any advice on what to look for would be helpful. Thanks.

[mdsnmss]

I'd start by making sure the forwarders are connecting to the indexers, not just the deployment server.

index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp

You should see connections from the forwarders and if they are connected you should also be able to search _internal for each host since they should be forwarding internal data. If they aren't connected I would check your outputs deployed to the forwarders.

If they are connecting, just not sending the Windows data you are expecting I would check the monitor on the forwarder. On a forwarder you can run "./splunk list monitor" to see which files it is trying to monitor and forward. If what you are looking for isn't listed I'd check the deployment server for the app that defines the inputs and make sure the app is set to trigger a restart when updated. It could be possible the app was deployed but Splunk was not restarted and the forwarder has not yet loaded in the inputs.

[ddrillic]

You better start with I can't find my data!