topic Re: Monitor index activity in Dashboards & Visualizations

Monitor index activity

x3ncrypt — Thu, 13 Jan 2022 04:06:54 GMT

Hi there, I would like to monitor indexes that have not been active for more than 24hrs+ and display the names of the indexes in a table as well as the last received activity. Thanks

Re: Monitor index activity

isoutamo — Thu, 13 Jan 2022 07:00:21 GMT

there are many apps which you can use for this. Here is some of those and other information about missing events.

There are a lot of options for finding hosts or sources that stop submitting events:
Meta Woot! https://splunkbase.splunk.com/app/2949/
TrackMe https://splunkbase.splunk.com/app/4621/
Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarder_warningsSome helpful posts:
https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
https://www.duanewaddle.com/proving-a-negative/

You could easily add alert based on those.

r. Ismo

Re: Monitor index activity

SinghK — Thu, 13 Jan 2022 07:14:02 GMT

| tstats latest(_time) as latest where index=* earliest=-24h by host,index | eval status = if(latest > relative_time(now(),"-23h"),"Success","Faliure"), Time = strftime(latest,"%c")|table host,Tindex,time,status

this will do

or use apps in @isoutamo 's reply

Re: Monitor index activity

johnhuang — Thu, 13 Jan 2022 07:19:10 GMT

Mostly right, but you're going to miss index that is offline for 25h+.

Should search for a longer time span, something like this:

| tstats max(_time) AS _time WHERE earliest=-7d@d BY index
| where _time<now()-(24*3600)

Re: Monitor index activity

x3ncrypt — Fri, 21 Jan 2022 05:10:57 GMT

Is that the entire search string I will need to use? Thanks

Re: Monitor index activity

isoutamo — Fri, 21 Jan 2022 07:23:40 GMT

based on @johnhuang SPL some modifications. If you want to check towards all indexes what you have defined on your indexers and don't want to get data from all time with tstats you should use this

Just replace splunk_server=* with your indexers.

This just get list of your all indexes which are defined on indexers and then get events from the beginning of previous day.

r. Ismo