Forwarding indexed data from an old indexer to a new indexer (not clustered)

[awilly162]

Hello,

I am attempting to forward data from an older indexer to a new indexer so that I can decommission the server the old indexer currently sits on. These indexers are not currently clustered, and the old is set up to forward to the new (so the indexes are all mirrored), but this was only sending new data, not any of the previously indexed data on the old.

What are my options? Am I able to forcibly forward the old data to the new? Do I need to manually sync the old data and the new by passing the old buckets to the new indexer? Ideally I'd like to migrate the data over time (there's a fair amount), but in my research so far that doesn't appear feasible.

[gcusello]

Hi @awilly162 ,

as also @PickleRick and @isoutamo said, forwarding indexed data isn't a good idea because you pay twice your license.

if you need to recover your old data you have two options:

extract all the data and ingest in the new indexers, but it's a very long job because you have to separately extract data for index, sourcetype and host, and anyway, you pay twice your license.

The second option is an off line option: if you have different indexes names, you can manually copy (with stopped Splunk) the indexes from the old Indexers to the new one, remembering to copy also the indexes.conf file.

In this way you migrated all the data without paying twice license but you'll have two indexes until the old ones will end the retention period.

There's a third option: engage the Splunk Professional Servces, but it is a bit expensive.

Ciao.

Giuseppe

[PickleRick]

There is yet another option - you can convert your installation to a cluster (you can have a cluster without default replication so that you don't have multiple copies of the same data). You need additional machine for Cluster Manager and you need to convert your buckets to clustered ones but then you can rely on cluster mechanics to move the data out of the old indexer as you decomission it.

[PickleRick]

"Forwarding" means exactly what is happening - you receive an event on input, you send it to output(s). That's it.

"Mirroring" of already existing data would be achievable only in cluster setup. Without a cluster you can only migrate the data offline.

[isoutamo]

Hi

usually this should do when you are migrating to new server. There are couple of old posts how this can do e.g. https://community.splunk.com/t5/Installation/Upgrading-and-migrating-to-a-new-host-how-to-migrate-la...

I’m afraid that your new server have all same (named) indexes as old node have? Also they have starting index numbering from start (0 or 1). This leads that you could have equally named buckets in both nodes (at least hot buckets). For that reason you couldn’t do rsync without losing some events or you are in situation when splunk cannot start.

https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/HowSplunkstoresindexes#What_the_index_dir... see bucket names.

In theory there are probably two or three way to do it. 

Create a new clustered indexers add/migrate those both nodes there, then transfer all buckets into one node and then go back to single node or keep it as one node cluster.

Transfer old indexes into new node but change index names and later use both indexes when you are make queries and/or create eventtypes for those.

Transfer old indexes to new node and ensure that there haven’t been any bucket name collisions. You must rename those directories before transfer. You also must update <index name>.dat file to highest <local id>+1 before starting nod again.

It’s mandatory to test these scenarios before you do it with your production nodes! Also take offline backups before transfer, so you could rollback if/when needed.

I haven’t try any of these as I have always migrate the data into new node before switchover.

Also ensure that you have same splunk version on both node!

If I must do this I probably chose 1st option?

r. Ismo