find disconnected Universal Forwarder

[maede_yavari]

Hi,

I have installed Splunk Universal Forwarder on several Windows servers, and they send their Windows logs to the indexers.

All Windows logs are saved in the 'windows-index.' However, sometimes, some of the Universal Forwarders are disconnected, and I have no logs from them in a period of time.
How can I find which Universal Forwarders are disconnected? I must mention that the number of UFs is more than 400.

[gcusello]

Hi @maede_yavari,

the best approach is having a lookup (called e.g. perimeter.csv) containing the lista of all UFs to monitor (at least one column: host).

Then you could run (e.g. every 15 minutes) a search like this:

| tstats count WHERE index=_internal BY host
| append [ | inputlookup perimeter.csv | eval count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

 If you don't wnt to have this lookup, you could use this search to run every 15 minutes:

| tstats count WHERE index=_internal earliest=-30d latest=now BY host _time
| eval period=if(_time<now()-900,"Previus","Last")
| stats dc(period) AS period_count values(period) AS period BY host
| where period_count=1 AND period="Previus"

this second solution has the advantage that you don't need to maintain the lookup but gives you less control because you don't check servers that aren't sending logs from 30 days and it's more heavy.

Ciao.

Giuseppe

[PickleRick]

Additionally, you can use one of several apps implementing such source tracking. For example - https://splunkbase.splunk.com/app/4621

On the other hand, you can use Forwarder Monitoring in Monitoring Console to see "lost" forwarders (but this relies on _internal logs from the forwarder, not on the actual "production" events forwarder from given UF)