Monitoring Splunk

How to troubleshoot why index size on disk suddenly jumped by 2.5TB overnight?

jbrandtelastica
New Member

On May 4th, the disk space used by our default index jumped from about 400G to about 3TB. This doesn't seem to be related to actual indexing-- we don't index anywhere near 2.5TB a day, and we didn't receive any license alerts.

I have used this search:

index=_internal source=*metrics.log | eval GB=kb/(1024*1024) | search group="per_sourcetype_thruput" | chart sum(GB) by series

...to look at the amount indexed per sourcetype for the days around the 4th, but there isn't any difference between them, and on none of these days did we index anywhere near that much data.

On the 5th, our disk usage returned to its old daily growth rate.

Is there a way to figure out what happened on the 4th, and why our index suddenly consumed so much disk space without warning?

0 Karma

grijhwani
Motivator

Search and tabulate the entries by punctuation, and see of there's a spike in any particular pattern.

Tabulate by hour, and look for the start of the increase.

Find the source of the anomaly.

The answer to your question is in the very data itself...

  • One thing to look out for is that you're not doing some weekly rotation and ending up with all the renumbered rotated logs being reindexed. (If you're using log rotation you also need to implement appropriate black/whitelist rules.)
0 Karma

jbrandtelastica
New Member

I don't think it's actually stuff being indexed-- looking at the query for license volume doesn't indicate an increase in the amount of data indexed.

Could it be metadata or something?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...