topic Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment? in Deployment Architecture

How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

guilmxm — Sun, 01 Nov 2015 10:41:30 GMT

Hi,

For troubleshooting and alerting purposes, I would like to be able to monitor the number of current active artifact objects in the dispatch directory of our search heads ($SPLUNK_HOME/var/run/splunk/dispatch) in a search head cluster deployment.

As Splunk warns when there more artifacts than the default limits. I guess it should be able to retrieve the number of artifacts in internal Splunk logs, _internal, _audit, _introspection or | rest command ?

Could not find the good search yet, is it possible?

Thank you in advance.

Guilhem

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

emechler_splunk — Sun, 01 Nov 2015 21:20:22 GMT

I think this search leveraging 'rest' should do what you want - there are number of ways to further differentiate between running / completed jobs, etc. if you need to break that out.

| rest /services/search/jobs | stats count

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

guilmxm — Mon, 02 Nov 2015 00:48:51 GMT

Hi,

Thank you for your answer.
I've checked on a standalone instance, and indeed it reports the number of current artefacts in dispatcher.

I will check how this works in sh cluster, and if i can get the result for each search head with the rest command.
And will revert

Guilhem

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

guilmxm — Thu, 05 Nov 2015 19:26:10 GMT

After having checked in deployment architecture, i indeed get the number of artifacts on the search head the search were executed.

In your knowledge, is there a way to target all of our search heads with the | rest command ?

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

lguinn2 — Thu, 05 Nov 2015 19:29:16 GMT

Have you considered using the Distributed Management Console on a search head (must be outside the SHC) and making it the "search head of search heads"?

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

guilmxm — Thu, 05 Nov 2015 19:43:25 GMT

Hi,

Yes it is already the case.

We are monitoring our distributed deployment (4x cluster indexer, 4x sh cluster, deployment, Heavy and Universal forwarders) from the DMC which is deployed in the master node.

The DMC has access to every peer, can i get the number of artifacts from the introspection data ?

Currently we are facing a 6.2.6 bug that prevents the captain from cleaning correctly artifacts (SPL official case opened, fix expected 18th november), this reveals to us the importance of monitoring artifacts of sh nodes, and i would prefer doing from Splunk directly more that writing an sh script to count the number of objects in dispatch directories of sh nodes... 🙂

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

cjonestsi — Wed, 11 Nov 2015 19:48:02 GMT

https://answers.splunk.com/answers/322363/upgraded-to-626-in-search-head-cluster-shc-environ.html
Here's the workaround if you don't want to go to 6.3

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

guilmxm — Wed, 11 Nov 2015 20:19:50 GMT

Hi,

Yes, that's correct, and interesting.
We've opened a case, and support gave us that information about the upcoming fix (in 6.2.8)
Migrating to 6.3.x would be nice, but we're not yet ready to.

Thanks for your comment

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

guilmxm — Sun, 20 Dec 2015 14:52:48 GMT

Ended with a small sh script that reports the number of directories within the dispatcher of each search head and we're good 🙂

Re: How can I monitor the number of current artifacts (search jobs in dispatch) from Splunk internal logs in a search head clustering environment?

leticiamartello — Tue, 15 Oct 2019 17:55:50 GMT

How can I find the current active artifact objects in the dispatch directory by user?