How do I monitor Splunk's latest event timestamp or index count from another tool?

[chrismenke]

We recently had an issue where Splunk services were up and running, but new data wasn't being indexed. I'd like to capture data on the LATEST EVENT or INDEXED count with HP SiteScope and report it to a dashboard.

My first thought was to match against the LATEST EVENT timestamp from the default user landing page, but SiteScope can't parse the JS. No problem, I isolated the JSON and was able to send the request and retrieve a good response...for a while. But because the request is dynamic (current timestamp), I need to create a dynamic path and query string. Okay, I can do that in SiteScope, but if I manually update the values I get the following:

{"messages":[{"type":"FATAL","text":"Unknown sid."}]}

Is there another way to poll the LATEST EVENT timestamp over http? Alternatively, I could pull the INDEXED Counter and make sure it is greater than the previous run...

Here's the request which works in real time, but becomes stale soon after:

http://SPLUNK/en-US/splunkd/__raw/servicesNS/USER/search/search/jobs/rt_1453156700/results_preview?o...

[jplumsdaine22]

The following search will give you how long ago the last event got indexed.

index=* | head 1 | eval last_event_seconds_ago=now()-_indextime| table last_event_seconds_ago

You can monitor this is a few ways - via a rest API search, or aa saved alert/report etc.

I would probably send the values to your monitoring system every 10 minutes or so (or however fast you like) as a custom alert action. Then your monitoring system can alert you if that value grows more than what you expect.