I want to check if anyone has any experience on expanding your Splunk system. The below is my situation.
Now: I have one Splunk server that acts as an indexer as well as a search head.
Plan: I planned to expand my indexer to 2 and use the current indexer as my search head.
What is the best way for me to do this?
What are the configuration I need to toy with to achieve this?
Stand up your 2 new servers, so now you have original=sh1, idx1, and idx2.
1: Point all forwarders via outputs.conf to idx2 so that no new data is coming to sh1. 2: Point sh1 via outputs.conf to idx2. 3: Restart sh1 and idx2; stop idx1. 4: Move the Indexed data from sh1 to idx1. 5: Replace outputs.conf everywhere to point to BOTH indexers now. 6: Restart all 3 Splunk servers. 7: Profit!
Obviously, your old data now exists only on a single indexer but it will age out. If you decide to add a
Cluster Master, you can do a
Bucket Rebalance to spread the old data across both Indexers.
I'm glad you got some help from the awesome @nickhillscpl 🙂 If his answer solved your question, please don't forget to resolve the post by clicking "Accept" directly below his answer. This will help other users who are in a similar situation find this recommendation.
You may find it easier to continue using your existing indexer as an indexer.
If you plan to implement a cluster, once you have created your master, you can simply add your existing indexer (and a new one) as indexing peers. (Note, your old data will remain on the original indexer, and will not be replicated, so you should account for that in your storage requirements.
You can then create a new search head, and simply copy all of your existing apps and knowledge objects to the new SH.
This avoids the challenge of moving your old buckets around.
I disagree; you are neglecting to consider just what a dumpster-fire the Search Head specific configurations are in most do-it-yourself all-in-one situations. It is usually EXCEEDINGLY complex identifying all of the mis-located knowledge objects and moving them. Keeping the all-in-one as a Search Head is waaaaaaaaaaay easier.
Hi Thanks for the quick reply.
The new indexer has a high disk space, so m i right to set the new indexer as master and after which set them as peer.
Another thing, because my search head is now on a static ip that my client users are on. I can imagine that i need to:
1) change the ip for the old indexer to another new ip.
2) change the ip for the new search head (system) to the old ip.
3) change the ip of the forwarder to the new indexer cluster master ip (this part i m not very sure).
You should use dns names rather than IP address. (I could rant for hours on “why”) but I would take this opportunity to move your Splunk deployment to dns names, leave the IPs, and and take the minor pain of getting users to change bookmarks now, but that’s just me.