The scenario is like, when i run the API and SPlunk UI query together, the API is taking more time when compared to UI. But i get the samples output as same. But when i scheduled the API to run for every 1 hour and when i get the error count in terms for 5k. At the end of the day, when i run the splunk UI query for that particular time where it showed 5k, but for the UI it shows as less than 100. Due to this, it causes fault alerts generation to monitoring team.
Eg:
Splunk API call run for every 1 hour. Each hour the count is arround 4k.
End of the day, i run a query from UI for time slice of every 1 hour.. Now i get the count as less than 100 for each hour. Where as API output has logged as 4k for every hour.
... View more