Getting Data In

Has anyone tried using fusemount for frozen storage?

AGLbwa
Path Finder

So I'm about to try using Azure Blob Storage fuse-mounted (using blobfuse) as frozen storage, I'm wondering if anyone else has tried this (even with S3) and what the results were? I mean unless the semantics are horribly broken it should work, but the devil (as always) is going to be in the details. I'm doing this (initially) with one indexer in the cluster and will report back if no-one else has preceded me down this path into madness!

Fingers crossed and see you (hopefully) on the other side!

B-)

1 Solution

AGLbwa
Path Finder

Update: had that one node running for a fortnight with /opt/frozen fusemounted to Azure Blob Storage (using blobfuse) with no errors. Not all filesystem semantics are supported (timestamps can be hinky and du returns BS), but it's good enough for frozen (and to prove it, yes I did thaw some randomly selected data (on a different platform) and yes, I could search it). I've cut across other nodes in the cluster and am almost finished.

Hope this helps someone else deciding whether or not to tread the path to madness and eventual despair! (Would recommend!)

B-)

View solution in original post

AGLbwa
Path Finder

Final update and one massive caveat that I haven't had a chance to fully investigate. This setup works brilliantly except if you have a DNS failure. We had a failure of the primary DNS server and this meant that name resolution on the system was a crapshoot (possibly due to shitstemd name resolution) - this made the fusemounts unusable (need to raise with MS - filesystem operations DO NOT timeout), and exposed a bug in Splunk, (Splunk relies on the underlying filesystem to timeout, and if it doesn't neither will Splunk), which meant Splunk would hang coming up as it attempted to access frozen storage (but there were no logs to indicate this).

AGLbwa
Path Finder

Update: had that one node running for a fortnight with /opt/frozen fusemounted to Azure Blob Storage (using blobfuse) with no errors. Not all filesystem semantics are supported (timestamps can be hinky and du returns BS), but it's good enough for frozen (and to prove it, yes I did thaw some randomly selected data (on a different platform) and yes, I could search it). I've cut across other nodes in the cluster and am almost finished.

Hope this helps someone else deciding whether or not to tread the path to madness and eventual despair! (Would recommend!)

B-)

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf25, and our Community

Thank you to everyone in the Splunk Community who joined us for .conf25, which kicked off with our iconic ...

Data Persistence in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. What happens if the OpenTelemetry collector ...

Introducing Splunk 10.0: Smarter, Faster, and More Powerful Than Ever

Now On Demand Whether you're managing complex deployments or looking to future-proof your data ...