Splunk Search

How do I monitor Splunk's latest event timestamp or index count from another tool?

chrismenke
New Member

We recently had an issue where Splunk services were up and running, but new data wasn't being indexed. I'd like to capture data on the LATEST EVENT or INDEXED count with HP SiteScope and report it to a dashboard.

My first thought was to match against the LATEST EVENT timestamp from the default user landing page, but SiteScope can't parse the JS. No problem, I isolated the JSON and was able to send the request and retrieve a good response...for a while. But because the request is dynamic (current timestamp), I need to create a dynamic path and query string. Okay, I can do that in SiteScope, but if I manually update the values I get the following:

{"messages":[{"type":"FATAL","text":"Unknown sid."}]}

Is there another way to poll the LATEST EVENT timestamp over http? Alternatively, I could pull the INDEXED Counter and make sure it is greater than the previous run...

Here's the request which works in real time, but becomes stale soon after:

http://SPLUNK/en-US/splunkd/__raw/servicesNS/USER/search/search/jobs/rt_1453156700/results_preview?o...
0 Karma

jplumsdaine22
Influencer

The following search will give you how long ago the last event got indexed.

index=* | head 1 | eval last_event_seconds_ago=now()-_indextime| table last_event_seconds_ago

You can monitor this is a few ways - via a rest API search, or aa saved alert/report etc.

I would probably send the values to your monitoring system every 10 minutes or so (or however fast you like) as a custom alert action. Then your monitoring system can alert you if that value grows more than what you expect.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...