Getting Data In

What is the preferred way to migrate Splunk indexes onto new servers?

Explorer

Hi All -

We have a bunch of Splunk indexes in place. Our application is going to migrate to a new set of servers. And we need to make a decision whether to use same Splunk indexes for the data on new servers or create new indexes. We have to run the application on both old and new servers for a good amount of time.

We have 2 options -

1) Reuse the indexes and create a new sourcetype for data from new servers.
For example: index=myindex sourcetype=application /// This will have data from old servers
index=myindex sourcetype=application-new /// This will have data from new servers.. index name remains same

2) Create a new index altogether for data from new servers
For example: index=myindex sourcetype=application /// This will have data from old servers
index=myindex-new sourcetype=application /// This will have data from new servers.. index name remains same

Both will involve some amount of work related to saved searches, dashboards, etc. But what is the preferred way to do this? As I understand, creating new indexes is a little more work and difficult to maintain. But what is the better choice between two options.

Thanks,
Payal

0 Karma
1 Solution

SplunkTrust
SplunkTrust

If you want to be able to query data from both new and old servers separately, then there has to be change in the user queries/reports/dashboards. My suggestion would be to create either an eventtype OR a macro for distinguishing between new and old server data, and ask user to use appropriate eventtype/macro to query corresponding data. Keep the same index and sourcetype name. This way the definition on where to get data is available in one centralized location and once you decommission old servers, you could just update corresponding eventtype/macro to get rid of them, without changing the queries.

The eventtype/macro definition could include following search
For new servers: index=myindex sourcetype=application host=newserver1 OR host=newserver2...
For old servers: index=myindex sourcetype=application host=oldserver1 OR host=oldserver2...

View solution in original post

SplunkTrust
SplunkTrust

If you want to be able to query data from both new and old servers separately, then there has to be change in the user queries/reports/dashboards. My suggestion would be to create either an eventtype OR a macro for distinguishing between new and old server data, and ask user to use appropriate eventtype/macro to query corresponding data. Keep the same index and sourcetype name. This way the definition on where to get data is available in one centralized location and once you decommission old servers, you could just update corresponding eventtype/macro to get rid of them, without changing the queries.

The eventtype/macro definition could include following search
For new servers: index=myindex sourcetype=application host=newserver1 OR host=newserver2...
For old servers: index=myindex sourcetype=application host=oldserver1 OR host=oldserver2...

View solution in original post

Explorer

Thanks! We will try this approach!

0 Karma

Builder

If I understand correctly that you're setting up new app servers and not new Splunk servers, why isn't the host field sufficient for distinguishing between the old and new servers?

0 Karma

Explorer

Host can also be used to distinguish between the old and new servers. The only question is that we need this migration to be as seamless as possible for the end users - who may not know how to tweak their splunk queries, but just use those as provided.

0 Karma