Splunk Enterprise Security

Will my multi-site clustering setup in Splunk Enterprise result in data loss?

ashishebansal
New Member

I have 2 sites with Multi-site clustering enabled, with one site as 3 indexes, 15Tb disk each, and another site with 2 indexes, 30Tb disk each.

My license is 200Gb and data retention policy is 15Tb or 365 days. Can this result in loss of data ?

0 Karma
1 Solution

lakshman239
SplunkTrust
SplunkTrust

I would think you would need consider a few others as well.

  • It would be good if you could add another indexer in site to to make them both 3 indexer node cluster.
  • The data replication across clusters will be based on your site replication factors [ RF]
  • When you say you are ingesting 200GB/day, assuming you get 40% compression, your disk usage/year will be 200*0.4* 365 = 29.2 TB, so assuming you would operate at 12TB per site, you can only 24TB, so there could still be cases where you see older data getting rotated to archive folders/deleted.

View solution in original post

0 Karma

tsaikumar009
Explorer

here we have to consider other factors as well, what is your replication factor ?

If there is no replication factor, and you are only maintaining one replica, the total usage will be as follows,

daily:
raw data = 200GB
replica = 200GB

so total of 400GB of data if you are utilizing complete license on daily basis, per year it would be 144 TB. You only have 75 TB of total disk space hence this would result in data loss.

Consider your retention policy is only 15 TB or 365 days which ever comes first , then in that case also the retention policy effects only the searchable data, but even then the data will be lost once you hit 75 TB of the total disk usage.

0 Karma

lakshman239
SplunkTrust
SplunkTrust

I would think you would need consider a few others as well.

  • It would be good if you could add another indexer in site to to make them both 3 indexer node cluster.
  • The data replication across clusters will be based on your site replication factors [ RF]
  • When you say you are ingesting 200GB/day, assuming you get 40% compression, your disk usage/year will be 200*0.4* 365 = 29.2 TB, so assuming you would operate at 12TB per site, you can only 24TB, so there could still be cases where you see older data getting rotated to archive folders/deleted.
0 Karma

ashishebansal
New Member

Thanks Lakshman, this is exactly what I have listed as my explanation but still wanted the second opinion.

Thanks much for your explanation.

0 Karma

tiagofbmm
Influencer

Data will not be lost due to ingesting more data or having smaller capacity to ingest data. What will happen is data will be rolled out of Splunk faster. So if you are ingesting 200GB a day, after 1year you have 73TB of data, so a the older buckets will be rolled to frozen, either deleted (by default) or to be processed by a script you may want to run when the time comes for them to be rolled from cold to frozen.

Helped?

0 Karma

ashishebansal
New Member

Thanks for your answer, it does help.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...