Sourcetype Dashboard and/or alert

[bluemarvel]

need to build an reporting alert that will indicate which sourcetype has stopped as well indicate which server, is there a method in which I can merge the two alerts together, if not then what would be the best approach. (not using Metadata)

[somesoni2 · Accepted solution]

Try like this. Run this for last 24hrs OR last 7 days

| tstats max(_time) as recentTime WHERE index=* by host sourcetype | eval age=now()-recentTime | where age>PutYourThresholdValueInSecHere | eval recentTime=strftime(recentTime,"%+") | eval age=tostring(age,"duration")

[dbcase]

I think the eval statement should read

eval age=now()-recentTime

[bluemarvel]

ok I tried this with 45404 sec and added an index, nothing comes up.

tstats max(_time) as recentTime WHERE index=* by host sourcetype | eval age=now()-recentTime | where age>PutYourThresholdValueInSecHere

[bluemarvel]

I also keep getting the following: "Error in 'tstats' command: This command is not supported in a real-time search"

is there another alternative

[dbcase]

Try putting the values into a table so you can see the age numbers. That way you can gauge what your threshold should be (ballpark)

|tstats max(_time) as recentTime WHERE index=* by host sourcetype | eval age=now()-recentTime | table host, sourcetype, age

You shouldn't have to make this a real time search to be able to alert on it.

[bluemarvel]

right, I got it to work, thank you. So the next step converting the recent time in a more readable format I.E.: (ddmmyy)

[dbcase]

Try this

https://answers.splunk.com/answers/145251/how-to-convert-epoch-timestamp-to-readable-date-format.htm...

[somesoni2]

I've updated my answer to include the human readable formatting of recentTime and duration.

In one of the comment you mentioned trying to run this as real-time search. If you're planning to do that, I would suggest use the regular search instead with schedule more frequently (say every 5-10 mins or so) if necessary. Real-time searches are expensive and a schedule real-time search never ends and will keep holding on the precious resources.

[bluemarvel]

ah got it, thank you for that

[somesoni2]

It should. Thanks for pointing it out. It's corrected now.