Upgrade Indexer Cluster w/ out search Warnings on Search Head

[jordanking1992]

Hello,

I have an automated upgrade plan that does the following:

1. Puts the cluster in maintenance mode
   1. splunk enable maintenance-mode
2. Goes 1 by 1 on each of the 3 indexer peers and runs:
   1. splunk offline
   2. Extracts upgrade tar file to necessary location
   3. runs splunk start and accepts license and answers yes.
   4. repeats for the next peer
3. Disables maintenance mode.

I am trying to upgrade the peers without the end users seeing messages but unfortunately users see things like the following:

Unable to distribute to peer named X because peer has status=Down. Verify uri-scheme, connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available.

^ Even though the peer is Up according to the Cluster Master

Connection Refused for peer=X

^ Which seems like the search heads are sending queries or still have an established connection with the peer. I would expect the search head to know that a peer is down and not communicate with it till it indexes have been validates and deemed searchable.

Anyone have recommendation on making the indexer upgrade as seamless to the end user as possible?

Things tried:

adjusted the restart_timeout, quiet_period, and decomission_node_force_timeout on the cluster master

Thanks,

J

[manjunathmeti · Accepted solution]

You can quarantine an indexer server while you upgrade it. This prevents search heads from connecting to the indexer server for any new searches.

Check this for more info: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Quarantineasearchpeer 

If this reply helps you, an upvote/like would be appreciated.

[jordanking1992]

Hey manjunathmeti,

Thanks for the recommendation, however, it looks like quarantining the peers helped quiet the errors above but now the user sees the following:

One or more peers have been excluded from the search because they have been quarantined. Use "splunk_server=" to search the peers. This may affect search performance

Do you know of anyway to silence this message?

[jordanking1992]

Disregard. To silence that message, you can set the target option to "none" in messages.conf.

Thank you for the solution to my question.

Respectfully,

J

[manjunathmeti]

You are welcome!
Instead of setting target to none, you can set it to log as suggested in the messages.conf documentation.

[jordanking1992]

Hey manjunathmeti,

Have you used the 'target = log' or 'target = none' settings? Although the message.conf.spec file says they are keys that can be used, there are no examples of them used in any of the messages.conf default. I have the following in my messages.conf on all three search heads under '/opt/splunk/etc/system/local/messages.conf' and still get the warning messages on my search heads when a quarantined indexer restarts.

[DISPATCHCOMM:EXCLUDED_QUARANTINED_PEERS]
message = One or more peers has been excluded from the search because they have been quarantined. Use "splunk_server=*" to search these peers. This might affect search performance.
severity = info
target = log

Any ideas?

J

[manjunathmeti]

I didn't try this attribute. Did you try setting it to none? 

You can try using other attributes, roles, capabilities to show the warnings to specific users/roles.