Getting Data In

Shutting down splunk Indexers For Upgrade

paul_1994
Path Finder

I am in a situation where I need to migrate my the splunk indexes to a bigger drive. I was wondering what would be a good way of accomplishing this.

I guess my question is what is the best way to shutdown these servers and upgrade them one at a time? Will this cause any issues? What happens with The Universal Forwarders?

My environment consists of 2 Search heads and 2 indexers with several Universal forwarders sending logs.

  1. my concern is what happens when I shutdown an Indexer.Does all the new data just go to one Indexer?
  2. When upgrading each server is there a problem having this Server down for 2-3 hours?
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Queue whatever the forwarder would hold, which is normally only a few hundred or thousand events, then the forwarders would stop accepting data.

paul_1994
Path Finder

What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

If you are sending from Splunk forwarders, then yes, while one indexer is down, all new data will go to the remaining ones. Assuming that one indexer can handle the load, the downside of this is really just that your data for that period will be unbalanced, so if you search for any data collected during the time, it's all stored on one node, so one node does all the work of retrieving the data. Over time, both will balance out, and if it's for a relatively short period (a few hours) there is no long-term harm. Of course the other disadvantages would be that if your one remaining server stopped while you were upgrading, you would of course be unable to index at all, and during the upgrade, data on the down indexer will be unavailable (and so searches will return incomplete results) but that I think is and obvious consequence.

paul_1994
Path Finder

What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...