Archive
Highlighted

Move specific data from one index to another index

New Member

Hello,

I want to move specific data from one index to another index. I don't want to make a full copy of previous index i just need to copy some of sourcetypes to another index. Then i need to delete copied data in old index.

Could someone please provide the best approach to do this?
I've read about collect command after search query but i believe it didn't actually copy your data on hard drive.

And i i've read about splunk cmd exporttool but not so much. Could someone please clarify on this point? Version of splunk is 6.0.1

Tags (2)
0 Karma
Highlighted

Re: Move specific data from one index to another index

Splunk Employee
Splunk Employee

It's difficult to export indexed events. While it is very easy to move/duplicate buckets.
If you cannot reindex your data, then my recommendation is to copy the buckets and hide the events.

  • identify the buckets with the data you need.

You can check the time range of each bucket endEpoch and startEpoch and the command :

|dbinspect index=myindex | convert ctime(startEpoch) | convert ctime(endEpoch) 

and use this trick to find in which buckets are your data if you want specific events

index=myindex sourcetype=BBBBBB | eval bkt=_bkt |  stats count by bkt

The only issue is that your buckets will not shrink when the events are deleted (until the buckets roll out to frozen), so this will initially use more space than before.

Highlighted

Re: Move specific data from one index to another index

New Member

Thanks for the answer,

I haven't found the way to delete specific data from local disk completely. As far as i know "delete" command just removes this data from search and not from local disk. Command splunk remove index="index" will delete entire index completely.

Can i remove from local disk data which was deleted via splunk search command?

0 Karma
Highlighted

Re: Move specific data from one index to another index

Super Champion

No. You can't modify data once it has been indexed.

0 Karma
Highlighted

Re: Move specific data from one index to another index

Splunk Employee
Splunk Employee

The delete just hide results, and does not reclaim disk space.
What I was saying is that the space will be retrieved once the whole buckets reach the timerentention limit. (default is 6 years or 500GB, but can be tweaked)

0 Karma
Highlighted

Re: Move specific data from one index to another index

New Member

I've found one of your previous answers to similiar question: http://answers.splunk.com/answers/25174/how-to-exportimport-events-from-indexes.

I see that i will have problem with exporting specific data with this approach.

It's a pity that there is no script for this even with REST API present.

Do you know how extensively RAM on server will be used during export? I have a very bad server running Splunk.

0 Karma
Highlighted

Re: Move specific data from one index to another index

Builder

Hi @yannK

what about this query?

index=INDEXNAME | eval bkt=_bkt 
|  table index,source,sourcetype,host,bkt
| join bkt 
    [| dbinspect index=INDEXNAME | convert ctime(startEpoch) | convert ctime(endEpoch) 
    |  rename bucketId as bkt]
| table index,source,sourcetype,host,bkt,state,sizeOnDiskMB,startEpoch,endEpoch,splunk_server

Thanks.

0 Karma