How to monitor and index actively?

[hochit]

Hi,

I have problem with my Splunk indexing. I found an index haven't been running and updated for 2 days. But it has no any sign for me until I searched and found out. I still can't identify it's Splunk or system problem.

Other indexes are working well, the license index volume is not exceeded.

So what can I do to monitor the health of an index actively? Also, what kind of debug log that I should turn on or look into?

Please advice!

Thanks, Philip

[bwooden · Accepted solution]

Additionally, inside Splunk some index health checks are available in the Search app under the menu Status -> Index Activity.

You could also enable an alert for a saved search with which you verify indexing health.

---

something like:

index=_internal group=per_index_thruput source=*metrics.log NOT series=_* |
eval last_seen=now()-_time | stats max(last_seen) as seconds_since_seen by series |
rename series as index 

[Pawlub1]

Hi, 

I would like to use it as an alert, but a bit confused the trigger

index=_internal group=per_index_thruput source=*metrics.log NOT series=_* | eval last_seen=now()-_time | stats max(last_seen) as seconds_since_seen by series | rename series as index | where seconds_since_seen < 120

Specifically, a value for the 'seconds_since_seen', if most indices are about the 800 second range, I am not sure if a low value like 120 seconds going to cause false positives.

Any suggestions for a proper value to monitor indices would be greatly appreciated.

Cheers, Paul

[hochit]

Thanks bwooden! The search is what I wanted!

The outcome is I can see there's no "index=_internal group=per_index_thruput..." for an index, and I don't have idea why.

Anyway, it's great preventive procedure for me.

[tedder]

I have very active indexes, so I just go to manager->indexes and look at the most recent time.