topic How to merge multiple index into single index? in Splunk Enterprise

How to merge multiple index into single index?

jack_lai — Mon, 30 Jan 2023 16:18:17 GMT

Hi splunk god,

Have enquiry, i have an environment which heavyforwarder logs send to cluster indexer.
I need the below multi index merge into single index which is index_general.
Basically, when user search index_general and able to search all the logs contain in the three index.

1)Is this configuration feasible?

index_fw->index_general
index_window->index_general
index_linux->index_general

2)If yes, this configuration needs to be done on HF or Indexer?

3)if qns2 yes, which config file should be configured.

Re: merge multiple index into single index

manjunathmeti — Mon, 30 Jan 2023 06:38:53 GMT

hi @jack_lai,

1) Yes, this can be done. But there are 2 things to consider.
1.1. Searches will be slower as you move 3 index data to one.
1.2. Data size of index_general should be the sum of the data sizes of 3 indexes and data retention should be the maximum value of data retention values of 3 indexes.

2) You can update inputs.conf on forwarders to send data to index_general index. But this will work only for new data.

3) For existing data you can use the collect command to write data to the index_general index.

Re: How to merge multiple index into single index?

jack_lai — Tue, 31 Jan 2023 06:57:20 GMT

How about if i got 2 cluster environments for example:

HF1->HF2>Indexer1
HF1->HF2>Indexer2

For Indexer1, the indexer should be able to query as per norm with 3 index.
For Indexer2, the indexer should be able to query with index_general.

I have tried other option which props/transform from sourcetype with _MetaData:Index in HF1, but this method affects the existing index and logs flow to Indexer1 as well. Is there any alternative option or technically feasible?