Adding an Index in Distributed Setup

[msarro]

Greetings,
I am in the midst of setting up a distributed search deployment (currently one search head, and one indexer, but we'll be adding 4 more indexers). What is the best method of adding actual indexes to each of the indexers? Is it as simple as logging in to the search head and adding an index there? Or is it going to be more involved? As of now there are no configured indexes outside of the ones from the stock installation - I need to add about 6 for our data).

[Sqig · Accepted solution]

If you don't want to use Deployment Server (we don't for a variety of reasons), you can use the Splunk command line to add indexes to each indexer. Note that after you do this on each Indexer, you'll need to bounce it.

These are the values we use:
/opt/splunk/bin/splunk add index $INDEX -homePath /opt/splunk/data/$INDEX -coldPath /opt/splunk/data/cold/$INDEX -thawedPath /opt/splunk/data/thawed/$INDEX

Actually, this is part of a script that populates values. Note that this will not work without valide credentials either entered "live" as you run the command interactively or else use the format -auth $AUTH at the end of the command-line above. This "auth" part is going to vary based on your security requirements. It is obviously not advisable to use the credentials on a command-line since that can be viewed by others.

[gekoner]

This answer is going to assume you have a Deployment Server. If not, that is the first thing to do, plan and implement a Deployment Server for your environment.

The best way to create and manage your Indexes is - Create an App for your indexes. You can do this using the standard App stanza in serverclass.conf and deploymentapps location (directory) on your Deployment Server.
Read: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Updateconfigurations
This will give you a good idea about what you'll need to do. Then you can ask specific questions here.
If you log in to the Search Head and add an Index there it will only exist on that search head, which is obviously NOT what you want.

I'd suggest you read this, if you haven't. This explains how the Deployment Server works, as well as Index and Search Head roles in a distributed Setup.
http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Distributedoverview

[kristian_kolb]

Word of caution on this one. I am a great fan of the DS, but I hadn't tried pushing indexer confs until yesterday. Unfortunately I was a little bit lazy when configuring the indexes.conf going out as part of the app; "No need to specify hot/warm/cold/thawed paths, just go with the default"... WRONG!

Since the indexer did not know where to store the indexes, it promptly died on startup, making the DS useless for fixing the problem. So I had to go out to each indexer and manually delete the bad indexer-app (after correctly defining it on the DS, of course).

You learn a little each day.

/k

[gekoner]

OK, well in that case the answer is easy. But not much fun in terms of management.
Simply log into each Indexer WebGUI and create the Indexes. You will obviously just have to do it on each Indexer to avoid problems. By problems I mean you will get a yellow-bar at the top of your search window, when you search and index if it DOESN'T exist on each Indexer is all. Be sure to store them in the same directory path too.
The other answer is to create the directory structure on each Indexer, under $SPLUNK_HOME/var/lib/splunk/ (linux) or & restart splunk. This assumes u put your dbs in this dir.

[msarro]

Sadly I've read through the entire distributed deployment guide. Up until now the plan had been to avoid using the deployment server (learning curve we don't have time for). Any information about how to configure indexes after you've setup the peer relationship is absent in the guide - I was looking to manually configure them for now on each indexer. I wasn't sure if creating the peer relationship affected the manual creation of indexes.