Copying over data to another index for longer retention

[jpillai]

Hi all,

We are indexing different topics from our kafka cluster to an index say, index1. But we now have a requirement to retain a subset of those topics for longer period of time. Is there a way to implement this while we still get all the data into the same index?

I can think of the subset of topics that I need longer retention being searched, filtered out and collected to a new index. But this involves licensing if we want to retain the source/sourcetype and other fields I believe which is not practical for us.

We want to retain the original source/sourcetype etc and have the subset of topics that we need longer retention be copied over to another index, say index2, that has longer retention.

We also need the original copy in index1 as we have lot of dependant searches and alerts that use this index to search for the same data.

[P_vandereerden]

I believe that your scenario could be accomplished with Ingest Actions:
https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/DataIngest

This should support cloning data and applying different filtering rules and routing to the two streams.