Is there an automatic way in archiving data in frozen data to s3 glacier?

[ejmin]

Hi, does anyone have an experience archiving data in S3 Glacier using a script or any third party apps. I already know the steps in uploading files in S3 glacier using aws cli commands but this kind of configuration is manual. My goal is to automatically upload all the data incoming in frozen directory to S3 Glacier. like how the splunk forwarder works

[isoutamo]

Hi
Atlassian has done example cold2frozen python script which you could use. It copies buckets to your defined S3.
r. Ismo

[ejmin]

Yeah... I saw it on a google but I think it is for S3 only. Is it also applicable to S3 glacier because S3 and S3 glacier use different storage system. S3 is for quick access storage while S3 glacier are for low storage cost archiving

[isoutamo]

That’s good point. I haven’t looked if it can store also on glacier or not. But i think that at least it can modify quite easy to store glacier also or do it via S3 as temporary storage?
I need to check this later on.
r. Ismo

[ejmin]

Ok I will thank you in advance for that..

[isoutamo]

Just looked AWS glacier api and it shows that you cannot/should use it directly from cold2frozen scripts. I suppose that move first to frozen as S3 and then regularly to glacier as batch or using some other automated ways.
r. Ismo

[ejmin]

Hmm you have a point there, it is possible to do that but I think it will not be possible on our side because it will  have additional cost to our end which should not. Our estimated data are more or less than a Terrabyte and as the year goes it will increase 1 TB in a statistical manner. So our only choice is S3 glacier

[isoutamo]

If you are doing that migration couple of times per day and reserve 2-5 day spool for issues then I don’t see that S3 as a big financial issue. 1TB on glacier means 10-15Gb S3 storage.