Getting Data In

Copying over data to another index for longer retention

jpillai
Path Finder

Hi all,

We are indexing different topics from our kafka cluster to an index say, index1. But we now have a requirement to retain a subset of those topics for longer period of time. Is there a way to implement this while we still get all the data into the same index?

I can think of the subset of topics that I need longer retention being searched, filtered out and collected to a new index. But this involves licensing if we want to retain the source/sourcetype and other fields I believe which is not practical for us.

We want to retain the original source/sourcetype etc and have the subset of topics that we need longer retention be copied over to another index, say index2, that has longer retention.

We also need the original copy in index1 as we have lot of dependant searches and alerts that use this index to search for the same data.

 

 

Labels (4)
0 Karma

P_vandereerden
Splunk Employee
Splunk Employee

I believe that your scenario could be accomplished with Ingest Actions:
https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/DataIngest

This should support cloning data and applying different filtering rules and routing to the two streams.

Paul van der Eerden,
Breaking software for over 20 years.
0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...