Monitoring scheduled searches

[fbl_itcs]

Hi folks,

I want to monitor my scheduled searches, e. g. I need to know if a schedulded search run while an indexer was down, which could lead to uncomplete results.

Does anyone know where to get those information? I'm aware of the fact that there is a sourcetype=scheduler in the _internal logs, but in this enviroment here (100gb+ per day) there are only 15 searches with a status != success while searching over all time. It seems that a scheduled search still has the status=success even if one of the indexer was down, which is an unexpected behaviour for me.

Any thoughts on this?

Kind regards,
Felix

[hexx]

This is a bit experimental, but I think it should work nicely:

| rest /services/server/info
| eval present = "yes"
| fields splunk_server present
| append [rest /services/search/distributed/peers splunk_server=local
| rename peerName AS splunk_server
| fields splunk_server]
| stats first(present) AS present by splunk_server
| eval present = if(isnotnull(present),present,"no")

So, this search will first list all peers that respond to a simple REST API query against the /services/server/info endpoint. This tells us who is there.

Next, we issue a second search against the search-head's /services/search/distributed/peers endpoint to find out which peers we expect to respond.

Finally, we aggregate both result sets and create a "present" field that will be set to "yes" for any peers that we were expecting to return results and did, and to "no" for any peers that we were expecting to return results but didn't.

Of course, it's trivial to add a where clause to have this search only yield results when one or more peers are missing:

| rest /services/server/info
| eval present = "yes"
| fields splunk_server present
| append [rest /services/search/distributed/peers splunk_server=local
| rename peerName AS splunk_server
| fields splunk_server]
| stats first(present) AS present by splunk_server
| eval present = if(isnotnull(present),present,"no")
| where present=="no"


[hexx]

I think that in essence, what you are asking for is new functionality for the scheduler execution records kept in scheduler.log. My recommendation is to file an enhancement request via our support portal, detailing the problem you would like to see resolved.

[fbl_itcs]

I don't really get why the state field is available but no useful information are put in there. I mean if an indexer doesn't respond to a search the state shouldn't be "success", because this wouldn't be true at all.
Is there really no log entry that tells us if a scheduled search ran successful or not?

[fbl_itcs]

Thank you for your answer.

I don't really see where this helps with my problem. Let's say we fire a scheduled search and it takes 2 minutes to finish. With your search I could check that all indexer and search heads are up and running before and/or after my scheduled search. But what if an indexer reboots or misbehave during a search?
Further than that it seems to be not really practicable to run your search like every 15-30 seconds because I would have to run it before and after every scheduled search.