Splunk SmartStore and searchable Events when using S3 and Glacier

[edwinmae]

We are using Splunk Enterprise, using SmartStore (S3).

Example: 

Index A, with frozentimeperiodinsecs = 7776000 (~90 days)

I understood that the EBS basically contains the cached events (that are searched a lot), but all event objects are stored in S3, right?

--

Let's say I have lifecycle policy set for the bucket that contains all the splunk data, using a prefix for (Folder) 'index A', with S3 > S3 I/A (30 days) and S3 I/A > Glacier (60 days)

If the event has been moved to Glacier, is the splunk search still working for that event?

Will the object be deleted after 90 days, meaning the object will be in Glacier for about 30 days (with the lifecycle policy in mind) and then deleted?

I need to test this, but if there is already some POC or test being carried out by somebody

Thanks

[richgalloway]

Do NOT use lifecycle policies for SmartStore buckets.  Allow Splunk to manage the data itself.  Data moved to Glacier will not be searchable by Splunk.

If you want to archive your old data then you can use the coldToFrozenScript setting to copy the data to a different S3 bucket which you manage.  See https://community.splunk.com/t5/Getting-Data-In/Splunk-SmartStore-Do-warm-buckets-need-to-roll-to-fr... for a good discussion of how to do that. 

[dbenicio]

---

@richgalloway wrote:

Do NOT use lifecycle policies for SmartStore buckets.  Allow Splunk to manage the data itself.  Data moved to Glacier will not be searchable by Splunk.

If you want to archive your old data then you can use the coldToFrozenScript setting to copy the data to a different S3 bucket which you manage.  See https://community.splunk.com/t5/Getting-Data-In/Splunk-SmartStore-Do-warm-buckets-need-to-roll-to-fr... for a good discussion of how to do that. 

---

OK, but why "Data moved to Glacier will not be searchable by Splunk." ? Are there any changes in indexed data that justifies it not being restored later or something like that? In this case I had this kind of problem, although I'm not sure if it is related or not to my Glacier/S3 lifecycle disaster recover policies transitioning ("copying") data into cold storage. 

If someone has had this issue or can comment on any conflicts observed regarding smartstore and glacier, I would really appreciate it.

[richgalloway]

Data in Glacier is not searchable because it's not supported by Splunk.  The delay in moving data back from Glacier to S3 probably is a large part of why, but Splunk doesn't say.

[edwinmae]

There are lot of discussions about SmartStore, but it seems that nobody really knows how it should work. There should be some clear POC (whitepaper) done by Splunk itself what to do when you need e.g. to store Logs for 10 years with SmartStore, but you want to archive them after 1 year .. and then when needed to restore data from archive back to searchable .. in case of a security audit request, etc.