Why does host stop sending logs to Splunk?

[NDabhi21]

Use case has been prepared with help of Splunk article 

https://www.splunk.com/en_us/blog/tips-and-tricks/how-to-determine-when-a-host-stops-sending-logs-to...

| tstats latest(_time) as latest where index=* earliest=-24h by host
| eval recent = if(latest > relative_time(now(),"-5m"),1,0), realLatest = strftime(latest,"%c")
| where recent=0

However receiving multiple false positive alerts for the windows servers(index=windows).

what will reason behind this ? its slow logs ingestion or in real there is no events for the mentioned index/sourcetype.

[PickleRick]

There are many possible reasons.

Firstly, I'd verify if the events are properly getting ingested. If you have such "false positive", just do a simple search on "index=<myindex> host=<myhost>" with values from the supposedly falsely positive result from your search and get a real-time search for a short while to see if the data is getting properly indexed (that's one of the rare reasonable use-cases for real-time search BTW).

This way you'll see if the events are getting ingested and - what's even more important - if they have proper timestamps.

Typical reasons for alerts regarding ingestion delay include:

1) Bad timestamp parsing - for example, events getting ingested with bad timezone

2) Bad time settings on source host

3) Batch processing - for example, events originating in WEF in pull mode or WMI-based EventLog monitoring might come in batches.

[dural_yyz]

Your trigger time should be based upon your own network knowledge of how your hosts operate.

1) Maintenance Windows - should be considered what your acceptable down time per host is

2) Lag between ingestion and event time - investigate local issues and any scripted intervals on collection, not all inputs are real time ingestion

Outside of what I have posted you would need to be very specific about what your UF inputs configurations are like and which sources per host have the most issues.  Thus far your questions contains some ambiguity which is forcing me to make a best guess.

Hope this helps, if not feel free to add more specific information.

[NDabhi21]

I have verify with one of the host , there is no Maintenance window or no lag between host and UF/Indexer

And Tired with below queries but result for windows logs are same.  Whenever there is no event(gap), splunk is consider as no event and triggering alert/notable No event seen from the host .

New SPL Query
| tstats latest(_time) as latest where (index=windows source="*WinEve:Security") by host, index, sourcetype
| eval age=now()-latest
|search age > 86400
|sort age d
|convert ctime(lastTime)
|fields age,host,lastTime

New SPL Query 2
| metasearch index=windows source="*WinEve:Security"
| eval host=upper(host)
| stats count by host
| stats sum(count)

[dural_yyz]

Morning,

Your search is still using _time which assumes the props or source timestamp is accurate.  Try adding in the _indextime field and use some calculations from there.  These types of searches will help identify any timestamp issues in your data.

https://community.splunk.com/t5/Splunk-Search/How-to-compute-indextime-time-difference-average-with-...

[NDabhi21]

Hey Dural_YYz

Thanks for your response.

Can please suggest best approach to achieve mentioned use case for the windows servers .

Specially which all are web servers where events are not continues .