topic Re: Move specific data from one index to another index in Knowledge Management

Move specific data from one index to another index

Navern — Tue, 11 Mar 2014 14:51:55 GMT

Hello,

I want to move specific data from one index to another index. I don't want to make a full copy of previous index i just need to copy some of sourcetypes to another index. Then i need to delete copied data in old index.

Could someone please provide the best approach to do this?
I've read about collect command after search query but i believe it didn't actually copy your data on hard drive.

And i i've read about splunk cmd exporttool but not so much. Could someone please clarify on this point? Version of splunk is 6.0.1

Re: Move specific data from one index to another index

yannK — Tue, 11 Mar 2014 15:21:31 GMT

It's difficult to export indexed events. While it is very easy to move/duplicate buckets.
If you cannot reindex your data, then my recommendation is to copy the buckets and hide the events.

identify the buckets with the data you need.

You can check the time range of each bucket endEpoch and startEpoch and the command :

|dbinspect index=myindex | convert ctime(startEpoch) | convert ctime(endEpoch)

and use this trick to find in which buckets are your data if you want specific events

index=myindex sourcetype=BBBBBB | eval bkt=_bkt |  stats count by bkt

copy them to the new index (if the index was not empty, avoid bucket id duplicated, see http://answers.splunk.com/answers/30986/why-is-my-index-disabled)
enable the delete command in your role (http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Delete )
selectively delete the events you do not want on the new index, and the opposite on the original index.
verify

The only issue is that your buckets will not shrink when the events are deleted (until the buckets roll out to frozen), so this will initially use more space than before.

Re: Move specific data from one index to another index

Navern — Tue, 11 Mar 2014 15:28:39 GMT

Thanks for the answer,

I haven't found the way to delete specific data from local disk completely. As far as i know "delete" command just removes this data from search and not from local disk. Command splunk remove index="index" will delete entire index completely.

Can i remove from local disk data which was deleted via splunk search command?

Re: Move specific data from one index to another index

lukejadamec — Tue, 11 Mar 2014 15:37:24 GMT

No. You can't modify data once it has been indexed.

Re: Move specific data from one index to another index

yannK — Tue, 11 Mar 2014 16:44:01 GMT

The delete just hide results, and does not reclaim disk space.
What I was saying is that the space will be retrieved once the whole buckets reach the timerentention limit. (default is 6 years or 500GB, but can be tweaked)

Re: Move specific data from one index to another index

Navern — Tue, 11 Mar 2014 17:57:06 GMT

I've found one of your previous answers to similiar question: http://answers.splunk.com/answers/25174/how-to-exportimport-events-from-indexes.

I see that i will have problem with exporting specific data with this approach.

It's a pity that there is no script for this even with REST API present.

Do you know how extensively RAM on server will be used during export? I have a very bad server running Splunk.

Re: Move specific data from one index to another index

splunkreal — Tue, 22 Jan 2019 15:31:15 GMT

Hi @yannK

what about this query?

index=INDEXNAME | eval bkt=_bkt 
|  table index,source,sourcetype,host,bkt
| join bkt 
    [| dbinspect index=INDEXNAME | convert ctime(startEpoch) | convert ctime(endEpoch) 
    |  rename bucketId as bkt]
| table index,source,sourcetype,host,bkt,state,sizeOnDiskMB,startEpoch,endEpoch,splunk_server

Thanks.