topic Re: New and old indexers, how can I ingest old data on the new indexer? in Getting Data In

New and old indexers, how can I ingest old data on the new indexer?

splunked38 — Tue, 14 May 2019 16:57:16 GMT

Hi,

We've set up a new index cluster and is currently cloning the data between the existing and new indexers.
https://docs.splunk.com/Documentation/Forwarder/7.2.6/Forwarder/Configureforwardingwithoutputs.conf#Configure_data_cloning_on_a_universal_forwarder_with_outputs.conf

We:
- can't 'reset' the forwarder as this will affect the existing indexers
- can't copy the index across to the new indexers (different versions, etc).
- still have the original log files (one shot is an option but we're open to better suggestions)

Is there a way to ingest old data into the new cluster?

Thanks in advance

Re: New and old indexers, how can I ingest old data on the new indexer?

aromanauskas — Tue, 14 May 2019 19:28:39 GMT

Your options are plentiful of course. But you need to determine some other factors first.

If you can keep the old indexer cluster around till the data would just expire anyways then unless you have a need to move the data it may be easier to just point the SH/C to both clusters.

If you have the RAW data, just put it all in a new directory and set the UF to only forward that data to the new cluster. You just have to be sure to setup the outputs correctly.

If you don't have the RAW data, then as long as you have 4.2+ on the old indexer you can treat it as if it was a frozen backup. Just copy the db/{bucket}/rawdata/journal.gz to $SPLUNK_HOME/var/lib/splunk/defaultdb/thaweddb/{bucket}/rawdata/journal.gz

Make sure the bucket_id is unique and then run the rebuild command as noted in the docs.

https://docs.splunk.com/Documentation/Splunk/7.2.6/Indexer/Restorearchiveddata

Re: New and old indexers, how can I ingest old data on the new indexer?

splunked38 — Wed, 22 May 2019 09:17:07 GMT

Sorry for the delay, thanks for the answer.

I haven't had a chance to try this out but I believe we'll be going down restoring from the raw data. The putting the data in a new directory may be tricky as we use the source for some of our queries but we might be able to do that through some source rewriting/transforms.conf.