Index Volume Location On Search Heads

[Gregski11]

still a total newb here so please be gentle, on Microsoft Window 2019 servers we have an Index cluster and here's how the Hot and Cold volumes are defined on it:

• C:\Program Files\Splunk\etc\system\local\indexes.conf

[default]
[volume:cold11]
path = E:\Splunk-Cold
maxVolumeDataSizeMB = 12000000

[volume:hot11]
path = D:\Splunk-Hot-Warm
maxVolumeDataSizeMB = 1000000

that I can live with, but on our Search Heads here's how we point on the volumes, and this don't look right to me:

• C:\Program Files\Splunk\etc\apps\_1-LDC_COMMON\local\indexes.conf

[volume:cold11]
path = $SPLUNK_DB

[volume:hot11]
path = $SPLUNK_DB

should the stanzas on the Search Heads match the ones on our Indexers?

[gcusello]

Hi @Gregski11 ,

at first I never saw a production Splunk infrastructure based on Windows (at least labs), think to use Linux!

Then, don't put conf files in $SPLUNK_HOME\system\local becaue you cannot manage them using a Deployment server: it's always better to put them in a cstom app (called e.g. TA_indexers) containing at least two files:

• app.conf,
• indexes.conf.

Anyway, having an Indexer Cluster, you define Volumes and Indexes on the Cluster Manager.

Then volume isn't relevant for Search Heads (Clustered or not): they point to the Cluster Manager to know the active indexers (using Indexers Discovery: https://docs.splunk.com/Documentation/Splunk/9.3.1/Indexer/indexerdiscovery ) or directly to the Indexers: https://docs.splunk.com/Documentation/Splunk/9.3.1/Indexer/Indexerclusterinputs).

About volumes and indexes, you can put them in your indexes.conf file following the nstructions at https://docs.splunk.com/Documentation/Splunk/9.3.1/Admin/Indexesconf

Ciao.

Giuseppe

[sainag_splunk]

Yes, It is the best practice  to have consistent index configurations & definitions throughout the cluster.

Thanks @gcusello @PickleRick  for the good points. To further elaborate on this topic and provide more details, I'd like to add the following:

EDIT: I'd like to expand on my previous answer with some additional best practices:

1. Separate Index Definition from Storage Definition:
- It's typically best practice to keep these configurations separate.
- In a production environment, use a legitimate app for your main indexes.conf file, not the system/local directory .
- This ensures better manageability, version control, and consistency across your Splunk deployment.

2. Use Separate Apps for Configurations:
- Implement a base config methodology with different apps for different aspects.
- Create apps like:
a) org_all_indexes: For consistent index definitions across the deployment.
b) org_idxer_volume_indexes: For indexer-specific configurations.
c) org_srch_volume_indexes: For search head-specific configurations.

3. Flexibility and Scalability:
- This approach allows different storage tiers for indexers and search heads as needed.
- It maintains a consistent view of available indexes across the deployment while allowing for component-specific optimizations.

These practices will help create a more robust, manageable, and scalable Splunk infrastructure.

[PickleRick]

Actually, no.

The indexes definition must be consistent across indexers in a cluster. From the technical point of view you don't need indexes definition on search heads. But it's useful for defining role permissions in UI and for auto-completion in search so since you don't need to index data on SHs it's actually a common practice to define an app with indexes definition and distribute it across both layers. But since typically indexers differ in storage layout from SHs it's also a well established practice, good from maintainability perspective, to define one app with indexes themselves based on volumes (this app you can distribute without changes on both tiers) and another app with volume definitions.

That's the case in @Gregski11 's case and it's that's actually a sound practice and there's nothing wrong with it.

You can kinda compare it with CIM datamodels which are defined withn an app and which you don't touch and their `cim_something_indexes` macros which externalize configuration of your datamodels from their actual definitions.