Monitoring Splunk

We need to upgrade the hardware on 1 of our Indexer server. What is the best approach for downtime for this upgrade?

Gh0st_rid3r
Explorer

Our Splunk Indexer is under resourced. To match Splunk support's recommendations we need to add more RAM to it. We have a deployment server with 2 indexer & 2 search head. This upgrade will require about 30 minutes of downtime .
What's the best approach for the hardware upgrade?

Labels (1)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Based on the description of your environment, I'm presuming the indexers are not clustered.

1) Ensure all forwarders are load-balancing between the two indexers.

2) Stop the indexer.

3) Add RAM

4) Restart the indexer

While the indexer is down data will continue to be sent to the remaining indexer so no data should be lost.  Searches, however, will return errors about being unable to reach all indexers and possibly having incomplete results.  For that reason, you may want to plan the outage around any important scheduled reports.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Based on the description of your environment, I'm presuming the indexers are not clustered.

1) Ensure all forwarders are load-balancing between the two indexers.

2) Stop the indexer.

3) Add RAM

4) Restart the indexer

While the indexer is down data will continue to be sent to the remaining indexer so no data should be lost.  Searches, however, will return errors about being unable to reach all indexers and possibly having incomplete results.  For that reason, you may want to plan the outage around any important scheduled reports.

---
If this reply helps you, Karma would be appreciated.

Gh0st_rid3r
Explorer

Thanks @richgalloway . You are right. The indexers are not clustered as per my knowledge. Havent come across the cluster master in our setup. Just indexers,search heads & deployment server.

Thanks. Will follow the steps as you suggested 😀.

0 Karma

aasabatini
Motivator

HI @Gh0st_rid3r 


Put the cluster in maintenance mode, stop the cluster master, shut down the cluster master, perform the upgrade, restart the cluster master, restart splunk (note splunk may be set to auto start), exiting maintenance mode occurs automatically with restarts.

Maintenance mode does not persist across master restarts.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Usemaintenancemode

 

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
Tags (2)
0 Karma

Gh0st_rid3r
Explorer

@aasabatini wrote:

HI @Gh0st_rid3r 


Put the cluster in maintenance mode, stop the cluster master, shut down the cluster master, perform the upgrade, restart the cluster master, restart splunk (note splunk may be set to auto start), exiting maintenance mode occurs automatically with restarts.

Maintenance mode does not persist across master restarts.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Usemaintenancemode

 


Hi @aasabatini , But that applies to the cluster master setup. In our environment, We dont have a cluster master.  Just the primary & backup indexers as same as search head. A deployment server for the configuration as per my knowledge. But thanks anyway.

0 Karma

aasabatini
Motivator

Hi @Gh0st_rid3r 

 

sorry I supposed you have a cluster enviroment. anyway I'm happy you resolved your doubs.

Regards

Ale

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

Gh0st_rid3r
Explorer

Thanks mate.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...