There are more than 1000 UF Windows and Linux systems installed. It is a distributed environment with around 100 systems at each location, one indexer deployed, and each indexer connected to a search head.
Our next step is to verify that all the hosts have been configured properly and are reporting to the indexer.
In cases where a host does not have the source or sourcetype, we need to update the list to match host and not match host against the below lookup table.
Could someone please suggest the spl.
your question contains two items:
About the first problem, you could use the alert contained in the Monitoring Console that checks the missing Forwarders,
Otherwise, you could create a lookup containing the hosts in the perimeter to monitor (called e.g. perimeter.csv) and schedule an alert like this:
| metasearch index=_internal | eval host=lower(host) | stats count BY host | append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count] | stats sum(count) AS total By host | where total=0
This second way gives you more control but is heavier than the other because you have to manually maintain the lookup.
About the second part, how do you manage your forwarders? are you using a Deployment Server?
If yes, you have to check the deployed configurations, if not I hint to use it.
Anyway, you should check, one by one, your flows (identified by sourcetype) anche check if you have parsing errors.
To avoid them, try to use the standard addons for Windows and Linux.
Thank you for your reply,
as I said, if you deploy the standard Technical Add-Ons (TA) for Windows and Linux using the Deployment Server, you're sure that the deployed configurations are OK and that they are deployed to all the systems.
then you could run some simple search as
index=wineventlog | stats dc(sourcetype) AS dc_sourcetype values(sourcetype) AS sourcetype count BY host
so you're sure that you're receiving all the sourcetypes in wineventlog; the dc_sourcetype value immediately identifies if there are someone missing.
Then running the same search on the windows and the os indexes you're sure that all the sourcetypes are present.
Thank you, but we do not manage the UF using the deployment server, TA Add-on.
We have manually installed the UF and configured the log files, so the reason we want to verify each has all logs reporting to the indexer is that all the logs need to be configured correctly.
Forwarders installation must be manual, eventually using a script (for Linux) and a tool as Ansible (for Windows).
Recently two apps to upgrade forwarders were released but I didn't still used them.
But anyway for configurations, this isn't a good practice and it's a very heavy additional work for you, so I hint to plan the use of DS.