Monitoring Splunk

For all reported universal forwarders ( Linux and Windows) how do I verify the sources and sourcetype?

kagamalai
Explorer

 

Hi,

There are more than 1000 UF Windows and Linux systems installed. It is a distributed environment with around 100 systems at each location, one indexer deployed, and each indexer connected to a search head.

Our next step is to verify that all the hosts have been configured properly and are reporting to the indexer.

In cases where a host does not have the source or sourcetype, we need to update the list to match host and not match host against the below lookup table.

Could someone please suggest the spl.

 

sourcesourcetype
WinEventLog:SecurityWinEventLog
WinEventLog:ApplicationWinEventLog
WinEventLog:SystemWinEventLog
  
/var/log/haproxy/haproxy.loghaproxy
/var/log/audit/audit.logaudit
/var/log/maillogpostfix_syslog
/var/log/messageslinux_messages_syslog
/var/log/croncron

 

Thanks

Manickam

Labels (1)
0 Karma

gcusello
Legend

Hi @kagamalai,

your question contains two items:

  • reporting of connected/Not connected systems;
  • configuration checks.

About the first problem, you could use the alert contained in the Monitoring Console that checks the missing Forwarders, 

Otherwise, you could create a lookup containing the hosts in the perimeter to monitor (called e.g. perimeter.csv) and schedule an alert like this:

 

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count]
| stats sum(count) AS total By host
| where total=0

 

This second way gives you more control but is heavier than the other because you have to manually maintain the lookup.

About the second part, how do you manage your forwarders? are you using a Deployment Server?

If yes, you have to check the deployed configurations, if not I hint to use it.

Anyway, you should check, one by one, your flows (identified by sourcetype) anche check if you have parsing errors.

To avoid them, try to use the standard addons for Windows and Linux.

Ciao.

Giuseppe

0 Karma

kagamalai
Explorer

Thank you for your reply,

  • reporting of connected/Not connected systems;  --- I have spl to check and installed and not installed UF status.
  • configuration checks. What is the best way to check all three or 5 logs reporting to indexer hostwise? We need to identify the host if one of the logs is missing on any host. 

 

Thanks

Manickam

0 Karma

gcusello
Legend

Hi @kagamalai,

as I said, if you deploy the standard Technical Add-Ons (TA) for Windows and Linux using the Deployment Server, you're sure that the deployed configurations are OK and that they are deployed to all the systems.

then you could run some simple search as

 

index=wineventlog
| stats dc(sourcetype) AS dc_sourcetype values(sourcetype) AS sourcetype count BY host

 

so you're sure that you're receiving all the sourcetypes in wineventlog; the dc_sourcetype value immediately identifies if there are someone missing.

Then running the same search on the windows and the os indexes you're sure that all the sourcetypes are present.

Ciao.

Giuseppe

 

0 Karma

kagamalai
Explorer

Thank you, but we do not manage the UF using the deployment server, TA Add-on.

We have manually installed the UF and configured the log files, so the reason we want to verify each has all logs reporting to the indexer is that all the logs need to be configured correctly.

 

0 Karma

gcusello
Legend

Hi @kagamalai,

Forwarders installation must be manual, eventually using a script (for Linux) and a tool as Ansible (for Windows).

Recently two apps to upgrade forwarders were released but I didn't still used them. 

But anyway for configurations, this isn't a good practice and it's a very heavy additional work for you, so I hint to plan the use of DS.

Ciao.

Giuseppe

 

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!