Mystery Universal Forwarder

[reed_kelly]

This may sound silly, but we don't have the ability to see how some of our Universal Forwarders (UFs) are configured. They are running on AIX and sending their logs to a heavy weight forwarder (HWF) through a firewall port.

We do have access to the HWF. We know that data is flowing from these UFs to the HWF, because the HWF metrics.log shows group=tcpin_connections messages with the right hostnames and even showing os=AIX. These same messages occasionally show a _tcp_eps value greater than 1.

We have no access to the UFs without extensive effort. If these servers are sending data, then we need it. The data is encrypted, so we can't just sniff the data stream. Is there any way to see what indexes the UFs are attempting to use?

[gnovak]

If you search for these AIX hosts, do you have the "index" field as a choice on the left hand side? If not if you pick "View all fields" on the left is "index" a choice? If so, if you add it as a field to display, does it tell you what index they are using to store their data? Is that what you mean? Of can you only see the HWF as the host in splunk....(but you obviously see other hosts in the metrics log...)

[gnovak]

I would think it would go into the defaultdb if you don't specify a specific index where you want the data to go...Hmmmm

[reed_kelly]

The indexers are showing _internal messages and local unix messages from the HWF itself. I don't see messages from the AIX hosts. If I configure the HWF to index locally, then I see the same pattern. I get local messages and _internal messages from the HWF, but no apparent data from the AIX systems.

I can also see that new buckets are being created for the local unix index, _internal and the audit index. So, even though I see the occasional _tcp_eps > 1, I can't seem to find that data. Is it possible that the HWF is discarding data if it doesn't have the corresponding index?