Getting Data In

Where can I find information on how to track my indexing volume over time?

mctester
Communicator

I am trying to find a search for index volume over time for licensing tracking.

I need to search by index or by host or by source type.

I need to provide a line chart to show growth and projected license violation to show when we will need more licensing and were we could provide filtering to lower event collection.

I also need to learn how to do event filtering from the client to lower unnecessary event collection.

1 Solution

Mick
Splunk Employee
Splunk Employee

You can find both guidance and searches to investigate your indexed data volume here. There are a number of searches on that page that will break down your indexed volume by host, source, sourcetype and index over time. By editing the search, you can look at weekly, daily or hourly totals, depending on your needs.

As for filtering events to keep your volumes down, that is all explained in our documentation here. This is resource intensive and will impact the speed at which your Splunk instance can index data, so if possible I would implement any filtering at the app level, rather than relying on splunk to do it

View solution in original post

hartfoml
Motivator

Thanks Mick, I tried your recomendation and pulled up only parcal ansers. For some reason the _interanl records for each day were stored under only a few dates like this -- 9/29/10 12:01:15.000 AM 10-04-2010 00:03:57.890 INFO LicenseManager-Audit - Audit:[quotaExceededCount=1, lastExceedDate=1284181219, peak=26631181931, rolloverCount=116, totalCumulativeBytesAtRollover=780873525059, todaysBytesIndexed=179325960, licenseSize=10240][MCp8taTlG+OTQsYggKfV0oVaeoCO9dPKPEYgWaOsf8qw6YpLbJjsgwTXmASiPbv65YE662cFMxu4UNySTNzL1FDZR6AdO/YBN9SgRw/u4TJOfsWF9gBSaOjnFYRGa7qR8ZpzCO3nJtEP7XAA9xSz0ScCQDRpRVHVwgtvhPbXwQn9WOaVy5rmoZKtKn/RbkmauPpEJPQulBHOd+l5RXaI26Ej0JX+qt9tdLFP7wjhHqjv6+CwdXmvdl1yZTWDvqeXKVNDFvl0+OJ8raLe8hwLbJTNoI/1igCnE+2mCKKOirkvtR9b6Jg1HX6n8Mg+vYwA3k4b5YzVCkA9cMPVbE9Egw==] host=escman01 Options| sourcetype=splunkd Options| source=D:\SPLUNK\var\log\splunk\license_audit.log Options

this record was created on 10-4 but the timestamp on this record is 9/27/10

this makes it hard to search since several records are timestamped with the same date and only one record is showen

0 Karma

Mick
Splunk Employee
Splunk Employee

You can find both guidance and searches to investigate your indexed data volume here. There are a number of searches on that page that will break down your indexed volume by host, source, sourcetype and index over time. By editing the search, you can look at weekly, daily or hourly totals, depending on your needs.

As for filtering events to keep your volumes down, that is all explained in our documentation here. This is resource intensive and will impact the speed at which your Splunk instance can index data, so if possible I would implement any filtering at the app level, rather than relying on splunk to do it

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...