Splunk Enterprise

How to copy/forward logs weekly to frozen archive?

rewritex
Contributor

Hello,

I'm trying to figure out how to do 3 months of HOT/WARM/COLD indexing but copy/forward logs every week to my frozen archive located in a separate location. I'm trying to compensate for some issues we are having with our infrastructure uptime. 

Q: Does this make sense and is this possible? Could anyone provide examples or advice?
Q: Is there a difference is storage space used by sending data in weekly vs monthly(or every 90 days)?

Also, Splunk is installed into a Windows Environment.

Thank You,
Sean

Labels (3)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

OK. So you'd like to copy out warm/cold bucket?

It is possible and copying warm buckets is one of the proposed backup strategies.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Indexer/Backupindexeddata

But I must say I've never done it myself.

View solution in original post

0 Karma

rewritex
Contributor

Thank you for the reply. I guess I'm not asking my question correctly..... 

Policy
1) 90-day - searchable data (HOT/WARM/COLD)
2) 90-day - frozenTimePeriodInSecs = 7776000 (move data or if 3) is used, delete data)
3) ?? 7-days - Weekly Powershell script to back-up/copy logs to remote store

My question on for 3) to compensate for some infrastructure issues, I want to back-up the indexed data sooner then waiting for the 2) frozentimeperiodinsecs. This may not be a feasible idea or make logical sense but this is where I'm at, at the moment and trying to think through it.  I have setup an index cluster with servers on different network segments to help with single point of failures so I'm hoping I can just depend on the standard 2) frozentimeperiodinsecs policy to move data to frozen remote storage.

Thanks again,
Sean

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. So you'd like to copy out warm/cold bucket?

It is possible and copying warm buckets is one of the proposed backup strategies.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Indexer/Backupindexeddata

But I must say I've never done it myself.

0 Karma

rewritex
Contributor

Lol, I wasn't searching with the correct words "hot / warm buckets". Thank you for the assistance!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

I'm not entirely sure what you want to do, to be honest.

You want to have your normal hot/warm/cold lifecycle and then once a week move the buckets that have already rolled to frozen somewhere off-site? You can do that of course. After the buckets are rolled to frozen, they are no longer visible to splunk for searching so you can safely move them outside.

But the question is is that really what you want, because that gives you an external copy of _old_ data (the buckets that already "expired).

And in terms of disk usage, the amount of data that gets rolled to frozen over some period should be roughly the same regardless of the schedule. After all it depends on the amount of data ingested.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...