topic Re: Computer Stopped sending logs in Splunk Search

How to find computers which stopped sending logs

Nawab — Mon, 29 Jul 2024 11:54:37 GMT

I have a deployment where multiple computers are sending logs to a WEF server using WEF(windows event forwarding). I tried to map ComputerName field to host name field but failed to do so.

Now I want to create an alert if any of the computer is not sending logs to splunk. how can i do so.

The method defined by splunk is based on index,host and sourcectype field, which will remain same for all computers in our case.

Re: Computer Stopped sending logs

gcusello — Mon, 29 Jul 2024 11:00:55 GMT

Hi @Nawab ,

if you have a list of hosts to monitor, you could put it in a lookup (called e.g. perimeter.csv and containing at least two columns: sourcetype, host) and run a search like the following:

if you don't have this list and you want to check hosts that sent logs in the last weeb but not in tha last hour, you could run:

| tstats count latest(-time) AS _time WHERE index=* BY sourcetype host | eval period=if(_time<now()-3600,"previous,"latest") | stats dc(period) AS period_count values(period) AS period BY sourcetype host | where period_count=1 AND period="previous"

The first solution gives you more control but requires to manage the perimeter lookup.

Ciao.

Giuseppe

Re: Computer Stopped sending logs

isoutamo — Mon, 29 Jul 2024 17:30:19 GMT

Hi
this is answer from Community Slack

Slackbot
17:08
There are a lot of options for finding hosts or sources that stop submitting events:
Meta Woot! https://splunkbase.splunk.com/app/2949/
TrackMe https://splunkbase.splunk.com/app/4621/
Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarder_warnings
Some helpful posts:
https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
https://www.duanewaddle.com/proving-a-negative/
r. Ismo

Re: How to find computers which stopped sending logs

Nawab — Tue, 30 Jul 2024 07:03:25 GMT

The issue in my case is the field i am look at is computername instead of host.

below is the deployement.

All windows servers ----> forwarder server ----> splunk

in splunk host will be forwarder server i.e 1 instead of the backend servers sending data.

these queries work on host source sourcetype and index fields.

Re: How to find computers which stopped sending logs

gcusello — Tue, 30 Jul 2024 07:21:33 GMT

Hi @Nawab ,

to use computername instead host youcannot use tstats and the search is slower, so try this:

with perimeter.csv lookup

without lookup:

index=* | stats count latest(_time) AS _time BY sourcetype ComputerName | eval period=if(_time<now()-3600,"previous,"latest") | stats dc(period) AS period_count values(period) AS period BY sourcetype ComputerName | where period_count=1 AND period="previous"

Ciao.

Giuseppe

Re: How to find computers which stopped sending logs

gcusello — Tue, 30 Jul 2024 07:40:08 GMT

Hi @Nawab ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉