Deployment Architecture

Hadoop Data Roll and archiving replicated buckets

delappml_2
Explorer

I have an indexer cluster with a replication factor of 3. If I were to implement Hadoop Data Roll, would only one copy of each event be archived to Hadoop at freeze time, or would all three bucket copies be archived? I'm trying to find out if I can save in terms of raw archive storage costs by implementing HDR versus archiving frozen buckets to a set of NFS mounts.

1 Solution

rdagan_splunk
Splunk Employee
Splunk Employee

HDR will only copy 1 journal.gz (raw data from the bucket). Therefore, 3X Splunk bucket replication will not impact the storage on the Hadoop side.

View solution in original post

rdagan_splunk
Splunk Employee
Splunk Employee

HDR will only copy 1 journal.gz (raw data from the bucket). Therefore, 3X Splunk bucket replication will not impact the storage on the Hadoop side.

mattymo
Splunk Employee
Splunk Employee

HDR is by far the best archiving solution, unless you really want to write your own dedup logic (spoiler: you don't lol).

- MattyMo
Get Updates on the Splunk Community!

Unleash Unified Security and Observability with Splunk Cloud Platform

     Now Available on Microsoft AzureThursday, March 27, 2025  |  11AM PST / 2PM EST | Register NowStep boldly ...

Splunk AppDynamics with Cisco Secure Application

Web applications unfortunately present a target rich environment for security vulnerabilities and attacks. ...

New Splunk Innovations Enhance Performance and Accelerate Troubleshooting

Splunk is excited to announce new releases that empower ITOps and engineering teams to stay ahead in ever ...