how to determine what exactly is happening with rolling data to cold storage

[jamesdon]

Hello,

We recently experienced a problem where our storage for cold data went offline. We are interested in determining exactly what happened to the data at that time, as well as when running a splunk fsdsk.

Is there a search prebuilt for this? Which file in /opt/splunk/var/log/splunk/ would have this info?

Thank you,

Jim

[jamesdon]

OK, it took us a bit to figure this one out, but the results may help someone out in the future. It ends out that we stopped rolling data over to cold after making changes to indexes.conf a while back. Lessons learned:

First, the logs are in /opt/splunk/var/log/splunk/splunkd.log.

Second, do not put trailing comments in your config files, it will cause confusion and delay. Something like:

[stan]
config = this # Don't put a comment here, its trouble

Jim

[twkan]

This sounds like a nasty issue to have, and may be similar to what I have encountered recently. Did your data somehow rolled over nto the frozen stage, and you do not have a coldToFrozen script defined?

[jbsplunk]

There isn't a prebuilt search for this, and I am not exactly sure what you mean by your cold data went offline. I assume its remote storage of some sort, such as nfs? I would review splunkd.log for errors about the buckets. I would imagine you'd see a lot of inflight buckets in warm that never made it to cold.