Splunk Enterprise

How to copy/forward logs weekly to frozen archive?

rewritex
Contributor

Hello,

I'm trying to figure out how to do 3 months of HOT/WARM/COLD indexing but copy/forward logs every week to my frozen archive located in a separate location. I'm trying to compensate for some issues we are having with our infrastructure uptime. 

Q: Does this make sense and is this possible? Could anyone provide examples or advice?
Q: Is there a difference is storage space used by sending data in weekly vs monthly(or every 90 days)?

Also, Splunk is installed into a Windows Environment.

Thank You,
Sean

Labels (3)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

OK. So you'd like to copy out warm/cold bucket?

It is possible and copying warm buckets is one of the proposed backup strategies.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Indexer/Backupindexeddata

But I must say I've never done it myself.

View solution in original post

0 Karma

rewritex
Contributor

Thank you for the reply. I guess I'm not asking my question correctly..... 

Policy
1) 90-day - searchable data (HOT/WARM/COLD)
2) 90-day - frozenTimePeriodInSecs = 7776000 (move data or if 3) is used, delete data)
3) ?? 7-days - Weekly Powershell script to back-up/copy logs to remote store

My question on for 3) to compensate for some infrastructure issues, I want to back-up the indexed data sooner then waiting for the 2) frozentimeperiodinsecs. This may not be a feasible idea or make logical sense but this is where I'm at, at the moment and trying to think through it.  I have setup an index cluster with servers on different network segments to help with single point of failures so I'm hoping I can just depend on the standard 2) frozentimeperiodinsecs policy to move data to frozen remote storage.

Thanks again,
Sean

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. So you'd like to copy out warm/cold bucket?

It is possible and copying warm buckets is one of the proposed backup strategies.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Indexer/Backupindexeddata

But I must say I've never done it myself.

0 Karma

rewritex
Contributor

Lol, I wasn't searching with the correct words "hot / warm buckets". Thank you for the assistance!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I'm not entirely sure what you want to do, to be honest.

You want to have your normal hot/warm/cold lifecycle and then once a week move the buckets that have already rolled to frozen somewhere off-site? You can do that of course. After the buckets are rolled to frozen, they are no longer visible to splunk for searching so you can safely move them outside.

But the question is is that really what you want, because that gives you an external copy of _old_ data (the buckets that already "expired).

And in terms of disk usage, the amount of data that gets rolled to frozen over some period should be roughly the same regardless of the schedule. After all it depends on the amount of data ingested.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...