topic Re: How to copy/forward logs weekly to frozen archive? in Splunk Enterprise

How to copy/forward logs weekly to frozen archive?

rewritex — Wed, 23 Feb 2022 18:27:02 GMT

Hello,

I'm trying to figure out how to do 3 months of HOT/WARM/COLD indexing but copy/forward logs every week to my frozen archive located in a separate location. I'm trying to compensate for some issues we are having with our infrastructure uptime.

Q: Does this make sense and is this possible? Could anyone provide examples or advice?
Q: Is there a difference is storage space used by sending data in weekly vs monthly(or every 90 days)?

Also, Splunk is installed into a Windows Environment.

Thank You,
Sean

Re: How to copy/forward logs weekly to frozen archive?

PickleRick — Wed, 23 Feb 2022 20:48:15 GMT

I'm not entirely sure what you want to do, to be honest.

You want to have your normal hot/warm/cold lifecycle and then once a week move the buckets that have already rolled to frozen somewhere off-site? You can do that of course. After the buckets are rolled to frozen, they are no longer visible to splunk for searching so you can safely move them outside.

But the question is is that really what you want, because that gives you an external copy of _old_ data (the buckets that already "expired).

And in terms of disk usage, the amount of data that gets rolled to frozen over some period should be roughly the same regardless of the schedule. After all it depends on the amount of data ingested.

Re: How to copy/forward logs weekly to frozen archive?

rewritex — Wed, 23 Feb 2022 21:20:57 GMT

Thank you for the reply. I guess I'm not asking my question correctly.....

Policy
1) 90-day - searchable data (HOT/WARM/COLD)
2) 90-day - frozenTimePeriodInSecs = 7776000 (move data or if 3) is used, delete data)
3) ?? 7-days - Weekly Powershell script to back-up/copy logs to remote store

My question on for 3) to compensate for some infrastructure issues, I want to back-up the indexed data sooner then waiting for the 2) frozentimeperiodinsecs. This may not be a feasible idea or make logical sense but this is where I'm at, at the moment and trying to think through it. I have setup an index cluster with servers on different network segments to help with single point of failures so I'm hoping I can just depend on the standard 2) frozentimeperiodinsecs policy to move data to frozen remote storage.

Thanks again,
Sean

Re: How to copy/forward logs weekly to frozen archive?

PickleRick — Wed, 23 Feb 2022 22:18:54 GMT

OK. So you'd like to copy out warm/cold bucket?

It is possible and copying warm buckets is one of the proposed backup strategies.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Indexer/Backupindexeddata

But I must say I've never done it myself.

Re: How to copy/forward logs weekly to frozen archive?

rewritex — Wed, 23 Feb 2022 22:27:05 GMT

Lol, I wasn't searching with the correct words "hot / warm buckets". Thank you for the assistance!