How to set up Indexes on Indexers

[robertlynch2020]

HI

We have installed a SH and 4 INDEXERS(Non Clustered). We have installed our app to the SH only with our indexers=mlc_live and or datamodels.
We have set up the forwarders to send data to the INDEXERS, however the SH is giving us errors saying

"Search peer hp400srv_6000_INDEXER1 has the following message: Received event for unconfigured/disabled/deleted index=mlc_live with source="source::/net/dell429srv/dell429srv1/apps/QCST_RSAT_v3.1.43_SEC1/logs/traces/mxtiming_286120_dell429srv_80849.log" host="host::NICKNAME" sourcetype="sourcetype::MX_TIMING2".

So the INDEXERS dont know about the Index=MLC_LIVE, so 3 questions

Do i manually set up indexes on indexers?
How do i manage my APP on my SH, so changes get passed over to all indexers?
Should i use the Deployer to move changes I make to get pushed over to the INDEXERS, like datamodels changes etc...?

Cheers in advance
Rob

[jdhunter · Accepted solution]

Since your environment is not clustered, you will want to create the index on each indexer. You can do this via the UI or from the CLI. Look at the Wiki below:

Splunk Web:

1. In Splunk Web, navigate to Settings > Indexes and click New.
2. To create a new index, enter:
3. A name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".
4. The index data type. For event data, click Events. This is the default data type.
5. The path locations for index data storage: Home path. Leave blank for default $SPLUNK_DB//db Cold path. Leave blank for default $SPLUNK_DB//colddb Thawed path. Leave blank for default $SPLUNK_DB//thaweddb
6. Enable/disable data integrity check.
7. The maximum size of the entire index. Defaults to 500000MB.
8. The maximum size of each index bucket. When setting the maximum size, use auto_high_volume for high volume indexes (such as the main index); otherwise, use auto.
9. The frozen archive path. Set this field if you want to archive frozen buckets. For information on bucket archiving, see Archive indexed data.
10. The app in which the index resides.
11. The tsidx retention policy. See Reduce tsidx usage. For more information on index settings, see Configure index storage.
12. Click Save.

CLI:
Edit indexes.conf

To add a new index, add a stanza to indexes.conf in $SPLUNK_HOME/etc/system/local, identified by the name of the new index. For example:

[newindex]
homePath=
coldPath=
thawedPath=
...
For information on index settings, see Configure index storage and the indexes.conf spec file.

Note: User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".

You must restart the indexer after editing indexes.conf.

https://docs.splunk.com/Documentation/Splunk/7.3.1/Indexer/Setupmultipleindexes

[richgalloway]

This is a good answer. I would add a strong recommendation to put your indexes.conf file into a custom app (myorg_all_indexes, for example) and install that app on all indexers. This helps avoid errors from making changes manually.

[jdhunter]

Excellent point Rich! Since he isn't running a clustered environment, he could use a Deployment Server to deliver his custom app to the indexers and avoid having to move it to each.

[robertlynch2020]

Hi

It is looking like i will use the Deployment Server
I will give it a go and get back. Just to ask one more question.

Will the Deployment Server be able to push out real time updates, as we update the APP in production daily. So we need to push out updates to data-models specifically.

Thanks for the help to all

Rob

[richgalloway]

Data models are stored on search heads, not indexers. The data saved by DMs is stored on the indexers.

[robertlynch2020]

Hi

Thanks for your help

We are seeing that the DataModels are stored on the Indexers not the search heads.

/splunk/var/lib/splunk/mlc_live/datamodel_summary

Cheers
Rob

[jdhunter]

Clients will check in periodically and compare the app on the DS to their app. If there is a change, the client will download.

https://docs.splunk.com/Documentation/Splunk/7.3.1/Updating/Howdeploymentupdateshappen

Process for setting up the DS:

https://docs.splunk.com/Documentation/Splunk/7.3.1/Updating/Planadeployment