Splunk Enterprise Security
Highlighted

Will my multi-site clustering setup in Splunk Enterprise result in data loss?

New Member

I have 2 sites with Multi-site clustering enabled, with one site as 3 indexes, 15Tb disk each, and another site with 2 indexes, 30Tb disk each.

My license is 200Gb and data retention policy is 15Tb or 365 days. Can this result in loss of data ?

0 Karma
Highlighted

Re: Will my multi-site clustering setup in Splunk Enterprise result in data loss?

Influencer

Data will not be lost due to ingesting more data or having smaller capacity to ingest data. What will happen is data will be rolled out of Splunk faster. So if you are ingesting 200GB a day, after 1year you have 73TB of data, so a the older buckets will be rolled to frozen, either deleted (by default) or to be processed by a script you may want to run when the time comes for them to be rolled from cold to frozen.

Helped?

0 Karma
Highlighted

Re: Will my multi-site clustering setup in Splunk Enterprise result in data loss?

New Member

Thanks for your answer, it does help.

0 Karma
Highlighted

Re: Will my multi-site clustering setup in Splunk Enterprise result in data loss?

SplunkTrust
SplunkTrust

I would think you would need consider a few others as well.

  • It would be good if you could add another indexer in site to to make them both 3 indexer node cluster.
  • The data replication across clusters will be based on your site replication factors [ RF]
  • When you say you are ingesting 200GB/day, assuming you get 40% compression, your disk usage/year will be 2000.4 365 = 29.2 TB, so assuming you would operate at 12TB per site, you can only 24TB, so there could still be cases where you see older data getting rotated to archive folders/deleted.

View solution in original post

0 Karma
Highlighted

Re: Will my multi-site clustering setup in Splunk Enterprise result in data loss?

New Member

Thanks Lakshman, this is exactly what I have listed as my explanation but still wanted the second opinion.

Thanks much for your explanation.

0 Karma
Highlighted

Re: Will my multi-site clustering setup in Splunk Enterprise result in data loss?

Explorer

here we have to consider other factors as well, what is your replication factor ?

If there is no replication factor, and you are only maintaining one replica, the total usage will be as follows,

daily:
raw data = 200GB
replica = 200GB

so total of 400GB of data if you are utilizing complete license on daily basis, per year it would be 144 TB. You only have 75 TB of total disk space hence this would result in data loss.

Consider your retention policy is only 15 TB or 365 days which ever comes first , then in that case also the retention policy effects only the searchable data, but even then the data will be lost once you hit 75 TB of the total disk usage.

0 Karma