There is a search that runs every 30 minutes and normally it runs okay. But in few instances seemingly at random the events shows up as 0.
index=eventlog (host="TP-Servers-*" OR host="TP-Hosts-*") splunk_server="splunkns0" sourcetype="event.log" | top limit=38 host | stats count as alive_hosts
We grab the time frame to be -30m to -10m at time of running the search.
The system uses this to confirm connection towards the hosts and servers. Below shows that at one point it shows 0 events found.
But when we run the search again of the same time frame. We find there were events (this time it should be around 8000 events).
We've checked the python, schedule, and splunkd logs and found no errors.
The times this occurs is not the same every time. Each time we search back there are results to be found. Has anyone else met this type of issue? Or has any other direction we can check?
