3 Forwarders shown in Deployment Monitor, but can search events from only one system

[tonopahtaos]

Hi,

I configured Splunk to receive events on port 9997 (the default value). Then setup 3 forwarders to send events to it. The first forwarder is Windows universal forwarder. The rest 2 are Linux universal forwarders. After that, i can see all 3 forwarders from Deployment Monitor.

But only events from first forwarder can be searched. So, I can only search Windows events. For other 2 Linux events, they are not shown in search summary page (only one host is shown on the Search/Summary tab's "Hosts" section). I can see total KB 1000Kb and 880Kb respectively for these Linux machines from Deployment Monitor's UI ('All Forwarders' tab) so Splunk does get events from these Linux boxes.

Anybody had this kind problem before?

TIA

[mikelanghorst · Accepted solution]

I'll go out on a limb with a couple of assumptions. For the Linux inputs, are you using the Splunk for Unix/Linux app? When you search are you specifying any indexes?

The Splunk for Unix/Linux application will send all of it's data to index=os, but from the Search app, the default out of the box index you'll be searching will be default/main, so you wouldn't find any data, nor would the Search Summary page show any of this data by default.

Add the following to your search, or use the *Nix app page (which does it for you)
index=os