What will happen when if the external NAS storage is not available while rolling the data to frozen path?

[kvm]

Hello Folks,

We are using the NAS storage to store our Splunk Frozen data, I'd like to know, if in case the NAS storage is not available (let's say NAS server down due to reboot, or any internal network issue between Splunk and NAS), what will happen to the rolling data, would it hold in Splunk until the Frozen path come back online, or Splunk will just ignore and drop the data?

Your response much appreciated.

[PickleRick]

There is varying info about this but remember - if digging up old threads - that moving to frozen used to be done by external script, now it's done internally by splunkd if I remember correctly.

Anyway, it will probably depend on what you mean by "storage is not available" and how it presents itself within OS and splunk.

For Splunk the database directories are just that - directory paths. It's up to OS to make sure there is something "down there" to write into.

So - for example - if you have an archive mount point which wasn't mounted, splunk should happily write to the directory on the main filesystem, not waiting for the mount to happen.

I'm not sure how splunk would handle situation when the storage had been mounted but "went away" due to power problems, network outage, whatever. From the OS point of view that could result in a timeout on attempts to access the directory. I'm not sure how splunkd would handle that.

So the answer is not that straightforward.

[gcusello]

Hi @kvm,

I encountered this problem some years ago and it was a real problem, because if the destination drive isn't available, Splunk creates a new destination in the SPLUNK_DB path and if there's some bucket to move from Warm to Cold, moves them in this wrong destination, then it's difficoult to recover those data.

Maybe i'ts possible to copy the rolled buckets in the correct destination, but I didn't tried, ask to the Splunk Support.

Ciao.

Giuseppe

[kvm]

Actually we are planning a reboot of the NAS storage server, probably 10-20 minutes, during the time if any bucket rolling happens what will happen to that data, that's my requirement.