Getting Data In

Can you give me some advice on expanding my Splunk systems?

quahfamili
Path Finder

Hi all,

I want to check if anyone has any experience on expanding your Splunk system. The below is my situation.

Now: I have one Splunk server that acts as an indexer as well as a search head.

Plan: I planned to expand my indexer to 2 and use the current indexer as my search head.

What is the best way for me to do this?

What are the configuration I need to toy with to achieve this?

Tags (2)

woodcock
Esteemed Legend

Stand up your 2 new servers, so now you have original=sh1, idx1, and idx2.

1: Point all forwarders via outputs.conf to idx2 so that no new data is coming to sh1.
2: Point sh1 via outputs.conf to idx2.
3: Restart sh1 and idx2; stop idx1.
4: Move the Indexed data from sh1 to idx1.
5: Replace outputs.conf everywhere to point to BOTH indexers now.
6: Restart all 3 Splunk servers.
7: Profit!
0 Karma

woodcock
Esteemed Legend

Obviously, your old data now exists only on a single indexer but it will age out. If you decide to add a Cluster Master, you can do a Bucket Rebalance to spread the old data across both Indexers.

0 Karma

ppablo
Retired

Hi @quahfamili

I'm glad you got some help from the awesome @nickhillscpl 🙂 If his answer solved your question, please don't forget to resolve the post by clicking "Accept" directly below his answer. This will help other users who are in a similar situation find this recommendation.

Thanks!

0 Karma

nickhills
Ultra Champion

You may find it easier to continue using your existing indexer as an indexer.
If you plan to implement a cluster, once you have created your master, you can simply add your existing indexer (and a new one) as indexing peers. (Note, your old data will remain on the original indexer, and will not be replicated, so you should account for that in your storage requirements.

You can then create a new search head, and simply copy all of your existing apps and knowledge objects to the new SH.
This avoids the challenge of moving your old buckets around.

If my comment helps, please give it a thumbs up!

woodcock
Esteemed Legend

I disagree; you are neglecting to consider just what a dumpster-fire the Search Head specific configurations are in most do-it-yourself all-in-one situations. It is usually EXCEEDINGLY complex identifying all of the mis-located knowledge objects and moving them. Keeping the all-in-one as a Search Head is waaaaaaaaaaay easier.

0 Karma

quahfamili
Path Finder

Hi Thanks for the quick reply.

The new indexer has a high disk space, so m i right to set the new indexer as master and after which set them as peer.

Another thing, because my search head is now on a static ip that my client users are on. I can imagine that i need to:

1) change the ip for the old indexer to another new ip.
2) change the ip for the new search head (system) to the old ip.
3) change the ip of the forwarder to the new indexer cluster master ip (this part i m not very sure).

0 Karma

quahfamili
Path Finder

Haha noted on the DNS, but generally is this the correct direction?

0 Karma

nickhills
Ultra Champion

Yes, sounds sensible. Good luck.

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

You should use dns names rather than IP address. (I could rant for hours on “why”) but I would take this opportunity to move your Splunk deployment to dns names, leave the IPs, and and take the minor pain of getting users to change bookmarks now, but that’s just me.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...