Can you help me with my index cluster set up?

[agentguerry]

Can anyone provide the steps to get an index cluster set up?

Splunk Docs seems to jump around a lot and not provide an instructional set up.

From what I gather, and what I have done is:

• Build out 3 Splunk servers

• Set up the first Splunk server as my master, setting my RF as 2, and my SF as 2.

• Set up Splunk box 2 and 3 as peers.

When viewing on my master, the "index clustering" page in the interface, i see that I have green checks, and that I have 2 peers searchable, and 3 indexes searchable (_audit, _telemetry, and _internal.

I think this is the correct way.

I have a couple of questions:

1. How do I go about adding another index to be searchable, such as If I wanted to monitor /var/log/messages?

2. Should my Universal Forwarder on Linux be pointing towards the master node, or does it point to my 2 peer nodes?

3. Do I have to go to each Splunk server, navigate to "Settings > Indexes", and create my "messages" index on each one?

Thanks!

[sarif_splunk]

Where you'll point your UFs depends on what approach you go ahead with.
- You can use indexer discovery feature as described here
https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/indexerdiscovery

• You can configure your UF's app to have a list of all your indexers mentioned in your inputs.conf file.

I would suggest to have minimum of 3 indexers. This protects you in case of an indexer failure.
Let's say you set your Replication Factor/Search Factor to 2. That means all the data you're ingesting would live on both indexers. In a situation where one of your indexer dies then the other one becomes a single point of failure. You'll also see Cluster Master complaining etc.. Having a minimum of 3 indexers would give you a lot of breathing space even if one of your indexer dies. Splunk would copy all the buckets (data) which was on the indexer which has died on the remaining 2 indexers.
Cluster Master can live on a light machine. A VM is ideal!

[woodcock]

1: How do I go about adding another index to be searchable, such as If i want to monitor /var/log/messages?
A1: You have a very jumbled question. Indexes that are defined on your indexes are searchable. There is no such thing as an unsearchable index (although you can configure what indexes users and roles can search in the access settings). If you would like to monitor /var/log/messages, you go to the node that has these, install a universal forwarder, setup outputs.conf to point to your indexers and setup inputs.conf with a monitor stanza pointed at that file.

2: Should my Universal Forwarder on linux be pointing towards the master node, or does it point to my 2 peer nodes?
A2: You can do either but I like to leave my CM as lightly loaded as possible so I always enumerate my indexers directly in outputs.conf.

3: Do I have to go to each splunk server, and navigate to "Settings > Indexes" and create my "messages" index on each one?
A3: Never use the GUI to do admin-level tasks. Always create a Splunk app from the CLI and deploy it to the indexers from the Cluster Master with a bundle push.

[adonio]

hello there,

i found splunk documentation very detailed and organized.
if you feel otherwise, dont hesitate to leave your comments at the relevant page/s and you will be approached and i am positive your concern/s will be addressed,

now for clustering.
in order to have an indexer cluster, you MUST have a search component, e.g. Search Head.
a minimum Indexer Cluster with replication, will have 4 machines
Cluster Master
Search Head
Indexers X2

in order to avoid complexity, use the GUI*, it is very self explanatory

to oyur other questions:
1. the question is unclear to me, in general, you need to add another configurations to indexers from Cluster Master, if you want to monitor locally on the indexer, you can also deploy this config form Cluster Master.
2. you will either point your forwarder directly to the Indexers or use Indexer Discovery, read here: https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/useforwarders
3. No, you should create an index in an app in the .../etc/master-apps/ directory on your Cluster Master, and distribute it to the Indexers, dont forget the repFactor configuration. see here: https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Indexesconf

hope it helps and good luck

[agentguerry]

So don't point the UF's to the master index, just the peers? (question # 2)

[sarif_splunk]

Where you'll point your UFs depends on what approach you go ahead with.
- You can use indexer discovery feature as described here
https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/indexerdiscovery

• You can configure your UF's app to have a list of all your indexers mentioned in your inputs.conf file.

I would suggest to have minimum of 3 indexers. This protects you in case of an indexer failure.
Let's say you set your Replication Factor/Search Factor to 2. That means all the data you're ingesting would live on both indexers. In a situation where one of your indexer dies then the other one becomes a single point of failure. You'll also see Cluster Master complaining etc.. Having a minimum of 3 indexers would give you a lot of breathing space even if one of your indexer dies. Splunk would copy all the buckets (data) which was on the indexer which has died on the remaining 2 indexers.
Cluster Master can live on a light machine. A VM is ideal!