Getting Data In

Why do I still get "Daily indexing volume limit exceeded" violation if we hard limit my overall index size to 500mb?

klops
Explorer

On this particular installation I don't care about historical data. So I set maxTotalDataSizeMB to 500mb.

However I still get hit by "Daily indexing volume limit exceeded" error when there are a lot of junk logging data coming to the indexer.

Tags (1)
0 Karma
1 Solution

Drainy
Champion

Because Splunk will drop old events from the bucket and fill it with new ones, you could have hundreds of gigs of data pass through a bucket of 500mb. You need to restrict what you are sending to the indexer or use the nullQueue to dump events before hitting the index

Oh and just to be clear, I assume you are using a free/trial license? If so that is limited to indexing only 500mb a day, so as explained above you are exceeding this which is causing the warnings. If you do it three times or more within a 30 day rolling period on a free or trial license then you will lose the ability to search for 30 days.

View solution in original post

Drainy
Champion

Because Splunk will drop old events from the bucket and fill it with new ones, you could have hundreds of gigs of data pass through a bucket of 500mb. You need to restrict what you are sending to the indexer or use the nullQueue to dump events before hitting the index

Oh and just to be clear, I assume you are using a free/trial license? If so that is limited to indexing only 500mb a day, so as explained above you are exceeding this which is causing the warnings. If you do it three times or more within a 30 day rolling period on a free or trial license then you will lose the ability to search for 30 days.

Drainy
Champion

Its quite easy to do and if you use the Universal Forwarder on your remote systems then you can do some more filtering before sending data on 🙂 Also if its helped then consider clicking the tick below the up and down arrow to the left of my answer, it will help others will similar problems in the future

0 Karma

klops
Explorer

thx for the reply. Yup, this is under the free license. I was trying to avoid going over the indexing limit by limiting the index size itself. But from your response it does look like I'll have to find ways to limiting the data input itself.

0 Karma
Get Updates on the Splunk Community!

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...

Splunk With AppDynamics - Meet the New IT (And Engineering) Couple

Wednesday, November 20, 2024  |  10AM PT / 1PM ET Register Now Join us in this session to learn all about ...

Building a Self-Service and Scalable Observability Practice

Thursday, November 14, 2024  |  11AM PT / 2PM ET Register Now Join us in this session and learn how Splunk ...