Re-ingesting partial logs from one host to another

[splunky1]

Our Splunk architecture is like

Two HFs pointing to Two internal Indexers and Two external Indexers.

Internal Indexers have different data and external indexers have different data (these indexers also receive data from other external HFs too) and HF's route the data correctly into the respective indexers.

We had a situation with fail over script which removed the outputs.conf file in both the HFs which resulted indexing the data locally into the HF and able to search this data in HFs but didn't go to indexers to search from search heads.

After we putting back the outputs.conf file the new data is going into the right indexers as intended but the data between is indexed into HF and lost in the indexers.

How can I re-ingest this data that's indexed in the HF into the indexers using the correct config.

I tried renaming the fish buckets folder and checked if that re-ingests the data but it only ingested small amount of data not everything.

I can still see data in my HF under $Splunk_home/var/lib/splunk/<index_name>/db/_raw 

What's the best way to re-ingest this data without manually moving the files into the indexers.

Thanks

[aasabatini]

Hi @splunky1 

Good Challange!

anyway some considerations

• if you have cluster indexer, move manually hte bucket is not great idea becuase you can create conflict with the bucket ID
• The source is present on the HF or come from to  a UF?

Options:

1.  Route data to splunk in syslog like is a third party system( cold be you will find to separate sourcetype)  https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Forwarddatatothird-partysystemsd
2. If you have access on  your monitor stanza (inputs.conf) and  try force to reindex data with this setting

crcSalt = <SOURCE>

https://splunkonbigdata.com/2018/10/13/how-crc-works-in-splunk/

• Try to summarize the logs with a schedule search (I know this is the more dirty option)