All Apps and Add-ons

1 of 12 universal forwarders not getting through

Techfrogger
Explorer

I've set up universal forwarders on 12 identical windows XP boxes and all are using the default port. Back on Splunk, using the Windows app, I can only see 11 of the 12 and I've verified that all settings are identical for the forwarder. Any idea where to look, or what to look at? Splunk server is installed on a Ubuntu box, version 12.04.1 LTS

0 Karma

MarioM
Motivator

what do you have in your culprit forwarder splunkd.log?

0 Karma

Techfrogger
Explorer

Still no answers, but now I have more news: this whole time I've been looking at Splunk via the Windows app. But if I ignore that and just search on the workstation name, thousands of items show up. So now it seems that Splunk IS getting the info I want but for some reason the Windows app for Splunk refuses to display info for this one workstation. How strange is that?!! Can anyone troubleshoot the windows app?

0 Karma

Techfrogger
Explorer

As to the question: what do you have in your culprit forwarder splunkd.log? I don't know the location of that log. Could you point me in the right direction?

0 Karma

Techfrogger
Explorer

Yes, that's correct: all 12 machines have identical configurations and are on the same subnet. And all data, from all hosts are sending to same index. Same configurations deployed to all agents. Really baffling.

0 Karma

lmyrefelt
Builder

All data, from all hosts coming to same index ? Same configuration/s deployed to all agents?

0 Karma

dart
Splunk Employee
Splunk Employee

And do you see those same sourcetypes for all 12 hosts?

0 Karma

Techfrogger
Explorer

WinEventLog: security, App and System

0 Karma

dart
Splunk Employee
Splunk Employee

What is the source and sourcetype for the data you are receiving?

0 Karma

Techfrogger
Explorer

I just ran netstat and it does show a connection to all 12 hosts. I then took a look at the firewall, ufw, and it shows activity from all 12 too. This is really baffling. I just installed some updates to the box and then rebooted, hoping that would be the end of it, but once again, all show up except the one in question.

0 Karma

dart
Splunk Employee
Splunk Employee

If you run netstat on the ubuntu box, does it show connections from all 12 hosts?

0 Karma

Techfrogger
Explorer

Sadly that's not the case. All are on the same subnet and the firewall they go through shows that all 12 are sending data. I assume the problem is on the Ubuntu server but that's where my understanding stops. Anyone know how to troubleshoot Ubuntu? Thanks everyone

0 Karma

ARothman
Path Finder

I'm unsure of what troubleshooting you have already tried, but I would start at the network. While understanding that you have 12 identical WindowsXP boxes, all using the default port, are they all on the same subnet with the same network rules applied to them? It sounds to me like this one you are having problems with may be behind a firewall that is blocking the forwarder traffic.

0 Karma
Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...