Splunk Enterprise Security

Why does host stop sending logs to Splunk?

NDabhi21
Explorer

Use case has been prepared with help of Splunk article 

https://www.splunk.com/en_us/blog/tips-and-tricks/how-to-determine-when-a-host-stops-sending-logs-to...

| tstats latest(_time) as latest where index=* earliest=-24h by host
| eval recent = if(latest > relative_time(now(),"-5m"),1,0), realLatest = strftime(latest,"%c")
| where recent=0

However receiving multiple false positive alerts for the windows servers(index=windows).

what will reason behind this ? its slow logs ingestion or in real there is no events for the mentioned index/sourcetype.

Labels (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

There are many possible reasons.

Firstly, I'd verify if the events are properly getting ingested. If you have such "false positive", just do a simple search on "index=<myindex> host=<myhost>" with values from the supposedly falsely positive result from your search and get a real-time search for a short while to see if the data is getting properly indexed (that's one of the rare reasonable use-cases for real-time search BTW).

This way you'll see if the events are getting ingested and - what's even more important - if they have proper timestamps.

Typical reasons for alerts regarding ingestion delay include:

1) Bad timestamp parsing - for example, events getting ingested with bad timezone

2) Bad time settings on source host

3) Batch processing - for example, events originating in WEF in pull mode or WMI-based EventLog monitoring might come in batches.

0 Karma

dural_yyz
Communicator

Your trigger time should be based upon your own network knowledge of how your hosts operate.

1) Maintenance Windows - should be considered what your acceptable down time per host is

2) Lag between ingestion and event time - investigate local issues and any scripted intervals on collection, not all inputs are real time ingestion

Outside of what I have posted you would need to be very specific about what your UF inputs configurations are like and which sources per host have the most issues.  Thus far your questions contains some ambiguity which is forcing me to make a best guess.

 

Hope this helps, if not feel free to add more specific information.

0 Karma

NDabhi21
Explorer

I have verify with one of the host , there is no Maintenance window or no lag between host and UF/Indexer

And Tired with below queries but result for windows logs are same.  Whenever there is no event(gap), splunk is consider as no event and triggering alert/notable No event seen from the host .

New SPL Query
| tstats latest(_time) as latest where (index=windows source="*WinEve:Security") by host, index, sourcetype
| eval age=now()-latest
|search age > 86400
|sort age d
|convert ctime(lastTime)
|fields age,host,lastTime


New SPL Query 2
| metasearch index=windows source="*WinEve:Security"
| eval host=upper(host)
| stats count by host
| stats sum(count)

0 Karma

dural_yyz
Communicator

Morning,

Your search is still using _time which assumes the props or source timestamp is accurate.  Try adding in the _indextime field and use some calculations from there.  These types of searches will help identify any timestamp issues in your data.

https://community.splunk.com/t5/Splunk-Search/How-to-compute-indextime-time-difference-average-with-...

 

0 Karma

NDabhi21
Explorer

Hey Dural_YYz

Thanks for your response.

Can please suggest best approach to achieve mentioned use case for the windows servers .

Specially which all are web servers where events are not continues .

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...