All Apps and Add-ons

How to create new index on index cluster?

HathMH
Explorer

Long post, newish to splunk, search strings are still a foreign language to me.

So I am tasked with incorporating azure gov into splunk. Splunk support recommended to use a particular app for microsoft cloud services. The app is easy enough to configure and whatnot. But having issue with creating an index for the app and ingesting into splunk.

We have the master node/deployment server, 8 indexers, 5 search heads, 2 heavy forwarders.

How do i create an index in a index cluster? 

I ask because the directions seem easy enough, however there are some hiccups. When I look at our indexes listed in splunk web, it does not match what is shown in the indexes.conf files. Which is in itself an issue.

These are the locations that I have found indexes.conf

$SPLUNK_HOME/var/lib/splunk  

 lists all my indexes and their dat files

$SPLUNK_HOME/etc/system/default/ 

the default files

$SPLUNK_HOME/etc/system/local/ 

has a listing of almost 80 indexes, but not all that are in the web portal search head, missing some of the sensitive indexes with naming conventions for systems like our txs and usr like txs_systemlog, usr-firewall, etc.

 


I went to our master node and the location $SPLUNK_HOME/etc/master-apps/_cluster/local/  to look at what the indexes.conf file says there...but its not present. Yet we obviously have indexes across our cluster.

So here are the issues:

1 - This prevents me from creating the needed index "usr-azure" as I do not where to put it.

2 - why are some indexes, like the sensitive ones, not listed in the conf files but are listed in the /var/lib/splunk/ ?

3 - Why is my master node web showing 48 indexes

HathMH_2-1659736827319.png

 

yet my indexers separately show 99 indexes?

HathMH_3-1659736865329.png

 

 

Additionally, another issue. I know we need to use CLI and edit the indexes.conf file for a indexer cluster, but I tried to do it via the web on indexer1, Settings >  Indexes (under Data), and I can click the New Index button. All is good, but when I get to the the App selection, it only lists all the apps.

HathMH_0-1659736238989.png

Whereas all the indexes listed show TWC_all_indexes

HathMH_1-1659736367241.png

 

Q4 - how do i get that for this app setting "TWC_all_indexes" for new index I am creating? I assume it has something to do with the index clustering and a setting on the master node. But I don't even see that option in the indexes.conf file.

 

 

 

 

Labels (2)
Tags (1)
0 Karma
1 Solution

gcusello
Legend

Hi @HathMH,

now I try to answer to your questions, but, I hint to follow a training course about cluster administration before to put your hands on an Indexer Cluster, to avoid additional problems or engage a Splunk Architect or a Splunk Professional Services!

Anyway, on Indexer cluster indexes.conf can be created only on Master Node and it can be located only in two folders:

  • $SPLUNK_HOME/etc/master-apps/_cluster/local/
  • $SPLUNK_HOME/etc/master-apps/<your_own_app>/local/

I hint to use the second location, in other words, create your own Add-On (called e.g. "TA_Indexes" or "TWC_all_indexes") and put this new app in $SPLUNK_HOME/etc/master-apps/ folder and deploy it to Indexers.

Remember, when you edit indexes.conf file, to use all the correct parameters, especially 

repFactor = auto

otherwise your index isn't replicated and you don't see it in Master Node's GUI.

So to create a new index, you have to create a TA containing indexes.conf and put all the information about this new index in it.

About the second question, you can see all the indexes.conf configurations, by CLI in one Indexer, using the btool command, described at https://docs.splunk.com/Documentation/Splunk/9.0.0/Troubleshooting/Usebtooltotroubleshootconfigurati...:

/opt/splunk/bin/splunk cmd btool indexes list --debug

so you'll be able to see all the indexes definitions.

In this way you'll see that probably there are other indexes.conf in some app you distributed.

About the third question, it means that you have many indexes not correctly created in your Indexer Cluster: in the Master Node are listed only Indexes that are replicated, not all indexes.

In other words, maybe there are indexes locally created on Indexers and not via Master Node or there are not replicated indexers (see the above parameter).

About your additional question, as I already said, you cannot edit indexes.conf on Indexers, only on the Master Node because editing isn't replicated to other Indexers and when the first update will be runned you loose every update.

About your fourth question, probably the TWC_all_indexes app was created by your architect to contain all the indexes and you can use it to insert the new index you need, this app surely is in $SPLUNK_HOME/etc/master-apps/_cluster/local/.

Then I don't understand what you mean with "I don't even see that option in the indexes.conf file", probably in this app there's the indexes.conf file to use in your cluster, you should check for all the indexes locally created or withour replication you have and then put them in this app's indexes.conf file.

At least, it isn't a good idea to have Master Node and Deployment Server on the same machine because they are two high work servers and you could have some problem.

It's a best practice to have a dedicated Deployment Server if it has to manage more than 50 clients, and when it shares the server, never with the master Node.

Ciao.

Giuseppe

View solution in original post

HathMH
Explorer

Thank you so much. I learned a lot from your reply. It appears we are not following some best practices such as the Master Node combined with the Deployment Server among others.

0 Karma

gcusello
Legend

Hi @HathMH,

now I try to answer to your questions, but, I hint to follow a training course about cluster administration before to put your hands on an Indexer Cluster, to avoid additional problems or engage a Splunk Architect or a Splunk Professional Services!

Anyway, on Indexer cluster indexes.conf can be created only on Master Node and it can be located only in two folders:

  • $SPLUNK_HOME/etc/master-apps/_cluster/local/
  • $SPLUNK_HOME/etc/master-apps/<your_own_app>/local/

I hint to use the second location, in other words, create your own Add-On (called e.g. "TA_Indexes" or "TWC_all_indexes") and put this new app in $SPLUNK_HOME/etc/master-apps/ folder and deploy it to Indexers.

Remember, when you edit indexes.conf file, to use all the correct parameters, especially 

repFactor = auto

otherwise your index isn't replicated and you don't see it in Master Node's GUI.

So to create a new index, you have to create a TA containing indexes.conf and put all the information about this new index in it.

About the second question, you can see all the indexes.conf configurations, by CLI in one Indexer, using the btool command, described at https://docs.splunk.com/Documentation/Splunk/9.0.0/Troubleshooting/Usebtooltotroubleshootconfigurati...:

/opt/splunk/bin/splunk cmd btool indexes list --debug

so you'll be able to see all the indexes definitions.

In this way you'll see that probably there are other indexes.conf in some app you distributed.

About the third question, it means that you have many indexes not correctly created in your Indexer Cluster: in the Master Node are listed only Indexes that are replicated, not all indexes.

In other words, maybe there are indexes locally created on Indexers and not via Master Node or there are not replicated indexers (see the above parameter).

About your additional question, as I already said, you cannot edit indexes.conf on Indexers, only on the Master Node because editing isn't replicated to other Indexers and when the first update will be runned you loose every update.

About your fourth question, probably the TWC_all_indexes app was created by your architect to contain all the indexes and you can use it to insert the new index you need, this app surely is in $SPLUNK_HOME/etc/master-apps/_cluster/local/.

Then I don't understand what you mean with "I don't even see that option in the indexes.conf file", probably in this app there's the indexes.conf file to use in your cluster, you should check for all the indexes locally created or withour replication you have and then put them in this app's indexes.conf file.

At least, it isn't a good idea to have Master Node and Deployment Server on the same machine because they are two high work servers and you could have some problem.

It's a best practice to have a dedicated Deployment Server if it has to manage more than 50 clients, and when it shares the server, never with the master Node.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...