So I'm about to try using Azure Blob Storage fuse-mounted (using blobfuse) as frozen storage, I'm wondering if anyone else has tried this (even with S3) and what the results were? I mean unless the semantics are horribly broken it should work, but the devil (as always) is going to be in the details. I'm doing this (initially) with one indexer in the cluster and will report back if no-one else has preceded me down this path into madness!
Fingers crossed and see you (hopefully) on the other side!
Update: had that one node running for a fortnight with /opt/frozen fusemounted to Azure Blob Storage (using blobfuse) with no errors. Not all filesystem semantics are supported (timestamps can be hinky and du returns BS), but it's good enough for frozen (and to prove it, yes I did thaw some randomly selected data (on a different platform) and yes, I could search it). I've cut across other nodes in the cluster and am almost finished.
Hope this helps someone else deciding whether or not to tread the path to madness and eventual despair! (Would recommend!)
Final update and one massive caveat that I haven't had a chance to fully investigate. This setup works brilliantly except if you have a DNS failure. We had a failure of the primary DNS server and this meant that name resolution on the system was a crapshoot (possibly due to shitstemd name resolution) - this made the fusemounts unusable (need to raise with MS - filesystem operations DO NOT timeout), and exposed a bug in Splunk, (Splunk relies on the underlying filesystem to timeout, and if it doesn't neither will Splunk), which meant Splunk would hang coming up as it attempted to access frozen storage (but there were no logs to indicate this).