topic Re: Search head and indexer connectivity in Splunk Search

Search head and indexer connectivity

Mag2sub — Tue, 22 Apr 2014 04:43:27 GMT

We have set up alerting searches with continuous scheduling from a search head with 2 peers
Soemtimes the search head loses connectivity with one of the peers
In this circumstance how does continuous scheduling work ..if it misses connectivity with one peer during a alerting search ...how do we safeguard against such circumstances ?

Apprecate inputs

Re: Search head and indexer connectivity

linu1988 — Tue, 22 Apr 2014 05:56:14 GMT

how do you come to know that? Does it show in the search result in the mail?

Re: Search head and indexer connectivity

Mag2sub — Tue, 22 Apr 2014 09:11:15 GMT

1 Im looking for a way to safeguard against idx to SH connectivity loss during search time...how does a search work in that context

2 Connectivity loss message can be seen in the idx that was disconnected to SH
im looking to see how we can ensure seraches are not incomplete because of the loss of idx--sh connection..does continuous scheduling help ?

Re: Search head and indexer connectivity

linu1988 — Tue, 22 Apr 2014 12:27:17 GMT

yes it will help if you have any error during the search. Use SOS app for monitoring the error trigger an alert. If it happens during a search it can't be avoided.

Re: Search head and indexer connectivity

Mag2sub — Wed, 23 Apr 2014 10:45:05 GMT

Unfortunately there is no error in the scheduler for scheduled search ...its says staus=success which is mileading

Re: Search head and indexer connectivity

linu1988 — Wed, 23 Apr 2014 12:09:44 GMT

status will always be success, if you can't get an instance or reproduce how do you want to test the alert? More often than not you will get an banner on splunkweb if there is an disconnection to search peer. then you know there must be some connectivity or indexer issue.

Re: Search head and indexer connectivity

MuS — Thu, 24 Apr 2014 07:13:54 GMT

Hi Mag2sub,

if your scheduled saved searches are configured to send alert emails you will get an email containing something like this:

-- Search generated the following messages -- 
Message Level: WARN
1. Unable to distribute to peer named <index-server> at uri https://<index-server>:<someportnumber>; because .....

In this case you would know about the condition you are describing.
You can also check out this answer the see a way to sent this kind of alert to a different recipient 😉

Best practice on the other hand would be to eliminate your cause for the errors.

hope this helps ...

cheers, MuS

Re: Search head and indexer connectivity

Mag2sub — Thu, 24 Apr 2014 09:16:08 GMT

We have a scripted alert and it does not get these from splunk ..it gives normal output ie normal arguments are passed and no error ...so i dont think this works for us

Re: Search head and indexer connectivity

MuS — Thu, 24 Apr 2014 10:27:40 GMT

pls hold the line and let me do some research. I know there is a way to do this as well in scripted alerts 😉

Re: Search head and indexer connectivity

MuS — Mon, 28 Sep 2020 16:27:48 GMT

ok, did you test SPLUNK_ARG_6 and / or SPLUNK_ARG_8 within your script? ARG_6 is an URL which could be loaded by your script and the parse the results or ARG_8 is the file system link to the result file which can be read by your script and be parsed for any errors.

Re: Search head and indexer connectivity

Mag2sub — Fri, 09 May 2014 14:49:42 GMT

$6 just hits the jobs results endpoint ..which does not indicate any error neither $8 ...