Splunk Search

How do I monitor Splunk's latest event timestamp or index count from another tool?

chrismenke
New Member

We recently had an issue where Splunk services were up and running, but new data wasn't being indexed. I'd like to capture data on the LATEST EVENT or INDEXED count with HP SiteScope and report it to a dashboard.

My first thought was to match against the LATEST EVENT timestamp from the default user landing page, but SiteScope can't parse the JS. No problem, I isolated the JSON and was able to send the request and retrieve a good response...for a while. But because the request is dynamic (current timestamp), I need to create a dynamic path and query string. Okay, I can do that in SiteScope, but if I manually update the values I get the following:

{"messages":[{"type":"FATAL","text":"Unknown sid."}]}

Is there another way to poll the LATEST EVENT timestamp over http? Alternatively, I could pull the INDEXED Counter and make sure it is greater than the previous run...

Here's the request which works in real time, but becomes stale soon after:

http://SPLUNK/en-US/splunkd/__raw/servicesNS/USER/search/search/jobs/rt_1453156700/results_preview?o...
0 Karma

jplumsdaine22
Influencer

The following search will give you how long ago the last event got indexed.

index=* | head 1 | eval last_event_seconds_ago=now()-_indextime| table last_event_seconds_ago

You can monitor this is a few ways - via a rest API search, or aa saved alert/report etc.

I would probably send the values to your monitoring system every 10 minutes or so (or however fast you like) as a custom alert action. Then your monitoring system can alert you if that value grows more than what you expect.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...