Splunk Search

How do I monitor Splunk's latest event timestamp or index count from another tool?

chrismenke
New Member

We recently had an issue where Splunk services were up and running, but new data wasn't being indexed. I'd like to capture data on the LATEST EVENT or INDEXED count with HP SiteScope and report it to a dashboard.

My first thought was to match against the LATEST EVENT timestamp from the default user landing page, but SiteScope can't parse the JS. No problem, I isolated the JSON and was able to send the request and retrieve a good response...for a while. But because the request is dynamic (current timestamp), I need to create a dynamic path and query string. Okay, I can do that in SiteScope, but if I manually update the values I get the following:

{"messages":[{"type":"FATAL","text":"Unknown sid."}]}

Is there another way to poll the LATEST EVENT timestamp over http? Alternatively, I could pull the INDEXED Counter and make sure it is greater than the previous run...

Here's the request which works in real time, but becomes stale soon after:

http://SPLUNK/en-US/splunkd/__raw/servicesNS/USER/search/search/jobs/rt_1453156700/results_preview?o...
0 Karma

jplumsdaine22
Influencer

The following search will give you how long ago the last event got indexed.

index=* | head 1 | eval last_event_seconds_ago=now()-_indextime| table last_event_seconds_ago

You can monitor this is a few ways - via a rest API search, or aa saved alert/report etc.

I would probably send the values to your monitoring system every 10 minutes or so (or however fast you like) as a custom alert action. Then your monitoring system can alert you if that value grows more than what you expect.

0 Karma
Get Updates on the Splunk Community!

New This Month in Splunk Observability Cloud - Metrics Usage Analytics, Enhanced K8s ...

The latest enhancements across the Splunk Observability portfolio deliver greater flexibility, better data and ...

Alerting Best Practices: How to Create Good Detectors

At their best, detectors and the alerts they trigger notify teams when applications aren’t performing as ...

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...