Installation

How do I find the source of indexed data volume?

rjlohan
Explorer

I am investigating Splunk for use with a new operational solution, but I seem to keep hitting the trial index limits and getting warned. The only data I've imported is the tutorial data, some small (<1MB) log files and some perfmon results using the 'Splunk for Microsoft Windows Infrastructure' app.

When I look at Settings->Indexes, the 'Current size' of all indexes is less than 200MB, but my license usage says I'm close to hitting 1GB.

How do I find where this licenses limit is calculated from? Perhaps some of my index samples are excessive. Since this is only a new (hours old) trial instance on a single machine, I'm concerned how the cost will actually pan out for a full live instance. These numbers aren't making sense to me right now...

Labels (2)
Tags (1)
1 Solution

jtrucks
Splunk Employee
Splunk Employee

To see your license usage, go to Settings menu, then Licensing. To further analyze the details, install the Splunk on Splunk (S.O.S.) app and use it to analyze your data sources and indexing/license use.

Also, the on-disk size of indexes is not the same size as your total ingested data volume due to compression. Generally it works out to roughly 50% disk usage compared to data ingested. However, this is an average that works out over time across multiple types of data sources. If you've ingested data that compresses poorly, you will have more disk used than 50%, or if you've ingested data that compresses well, you will have less disk used than 50%.

--
Jesse Trucks
Minister of Magic

View solution in original post

bhawkins1
Communicator

In Splunk 6, in the license manager /en-US/manager/search/licenseusage, the "Previous 30 Days" tab gives you the option to "Split By", for example, Source Type, which was sufficient for my usage. No need for additional apps.

0 Karma

jtrucks
Splunk Employee
Splunk Employee

To see your license usage, go to Settings menu, then Licensing. To further analyze the details, install the Splunk on Splunk (S.O.S.) app and use it to analyze your data sources and indexing/license use.

Also, the on-disk size of indexes is not the same size as your total ingested data volume due to compression. Generally it works out to roughly 50% disk usage compared to data ingested. However, this is an average that works out over time across multiple types of data sources. If you've ingested data that compresses poorly, you will have more disk used than 50%, or if you've ingested data that compresses well, you will have less disk used than 50%.

--
Jesse Trucks
Minister of Magic

rjlohan
Explorer

Thanks for this suggestion; the S.O.S indexing detail clearly shows a breakdown which is exactly what I needed. winnetmon looks to be the main culprit. Probably misconfigured that one for what I really needed.

0 Karma

Richfez
SplunkTrust
SplunkTrust

May I suggest the Fire Brigade app?

If you are on 6.x (I expect so given your description above), you'll need
Technology Add-on for Fire Brigade version 2
and
Fire Brigade version 2

It will take a day for it to generate information, but you should be able to find out more information then.

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...