Move logs to another Index

JHill — Wed, 12 May 2010 21:54:39 GMT

Is there any way to move log data to another index after it has already been indexed?

Example..

Windows logs were indexed into the main(default) index. I would like to move the data from that one host into another index.

Running Splunk 4.0.11

Re: Move logs to another Index

dwaddle — Thu, 13 May 2010 04:56:07 GMT

After data has been stored in an index, it can't be moved. It can be deleted using the delete operator and then you could reload it into the proper index. (If it came from a flat-file or other simple reloadable source).

Re: Move logs to another Index

Lowell — Thu, 13 May 2010 20:42:43 GMT

Yes. This can be done, the question is how badly do you need to move the data. As of Splunk 4.x, you can search multiple indexes at once, so this is often not a big deal. If however, you cannot find a way around moving your events from one index to another, than the following may be of help:

Events can be exported and imported using exporttool and importtool. Which is one way of moving events from one index to another, but this can be a fairly daunting task. Especially if you are trying to move a source or sourcetype that splunk has been indexing for a long period of time.

Essentially, these tools can be used to import and export the content of a bucket using a simple CSV file format. You can export the content of one bucket (from you existing index) and then import those events into a bucket (in your desired index). If your source/sourctype to be moved exists across multiple buckets in your source index, then you will have to deal with each bucket individually. (This is where it starts to get really ugly, time-consuming, and potentially dangerous to your data if you screw up...)

If you are looking to only move a few sources/sourcetypes, then you may be able to do so by simply indicating the source/sourcetype when calling exporttool, otherwise you may need to export all the events (using meta::all) and write a small CSV-processing script/utility of some kind to sift out the events you want to migrate to a different index.

For example, if you are trying to move the sourcetype WinEventLog:Application from the main (default) index to the os index, something like this could get you started:

splunk cmd exporttool defaultdb/db_1262807912_1262278800_6 /dev/stdout -csv sourcetype::WinEventLog:Application | splunk cmd importtool os/db_temp /dev/stdin

Of course at this point we have only copied the events. You still have to remove the events from the source bucket. After that, you would need to remove the events from your default index, and rename the db_temp bucket with an appropriate name (see the script on the wiki page linked below)

You may find some additional information here:

http://www.splunk.com/wiki/Community:Modifying_indexed_data_via_export_and_import

Update: As of Splunk 4.1.4(ish) the meta::all is not longer recognized by the exporttool command. Simply omitting that option should result in all events being exported.

Re: Move logs to another Index

peter_krammer — Tue, 25 Nov 2014 16:32:16 GMT

Is this still possible in Splunk 5 and 6?

topic Re: Move logs to another Index in Getting Data In

Move logs to another Index

Re: Move logs to another Index

Re: Move logs to another Index

Re: Move logs to another Index