Splunk Search

*Simple* daily volume tracking?

gowen
Path Finder

If I go into the License Manager, it shows me a simple progress bar of "Volume used today". For pool "auto generated pool enterprise", I see (for example) "Volume used today" of 3,389 MB / 10,241 MB (at 10:04 AM).

What I'm looking for is very simple: I want to know this value for the last N days. I want to know, at 23:59:59, I was at 9,506/10,241 on Monday, 10,506/10,241 on Tuesday, 10,103/10,241 on Wednesday, and so forth.

There are many answers addressing daily log volume, and none provide this simple level of answer. One actually suggests that this type of graph used to be available but no longer. I do have the Deployment Monitor app and the Splunk License Usage app installed, both are useful at seeing the breakdown of where logs are coming from, but don't give me this simple daily high water mark that I'm looking for.

It shouldn't take rocket science to be able to say to my boss "our daily usage has trended from 85% to 95% average capacity over the last two months, so give me money for more licenses."

Any help out there for simple summaries of daily volume?

Tags (2)
1 Solution

gowen
Path Finder

I've ended up with a suggestion from Tim Cashin, excellent Splunk sales rep. In the "Deployment Monitor" app, under "License Report" there is a table "Daily License Usage for Last 60 Days". This appears to be reasonably close.

I say "reasonably close" because you can view "By Indexer" or "By License Pool" and the two don't agree. Yesterday my two indexers were at 2.33 and 5.74 GB, but my license pool logged 13.81 GB. I have a 10 GB license and had no violation yesterday. License pool seems off by roughly x2, but only roughly (add up indexers to get 14.07, != 13.81).

It has been close enough to demonstrate the pattern of usage to my management, which was my goal.

View solution in original post

wrangler2x
Motivator

This search works very well for me, and I run it over the last 30 days, but you can pick any number of days, like last 7. I'll give you a day-by-day statistics report of volume used per day by license usage, and if you click on the Visualization tab you'll get a bar graph by default, and you can, of course, change it to a line graph (or whatever). Give it a whirl.

index=_internal source=*license_usage.log type="RolloverSummary" | eval _time= _time - 43200| bin _time span=1d | stats latest(b) AS b by slave, pool, _time | timechart span=1d sum(b) AS "Volume" fixedrange=false | fields  - _timediff -stacksz | foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 5)]
0 Karma

a1ay
Explorer

Will this search show the exact lincence use /day.. i mean including buffer/cap limit.

0 Karma

pkeller
Contributor

Just a question regarding the resetting of _time. What's the purpose of setting time backwards by 1/2 day?

0 Karma

gmriley
New Member

I tried this:
index=internal metrics kb series!=* group="per_index_thruput"| eval indexed_GB = (kb / 1024 / 1024) | timechart eval(round((sum(indexed_GB)),2)) as TotalGB fixedrange=t span=1d

Then went back to this: Status > Index activity > Indexing volume, splitting for one host and a given day.

The first says 1.99GBs, the second (when my indexes, not internals, are added up) says 1.65GBs.

SOS (at v4.3.5) is buggy and until we upgrade I can't use that. I have a command line that looks interesting but want a search I can save and configure email reports with. Is there a search that jibes with other tools?

0 Karma

sowings
Splunk Employee
Splunk Employee

You just want a total?


index=_internal source=*license_usage.log type=RolloverSummary | bucket _time span=1d | stats sum(b) AS bytes by _time

The RolloverSummary events occur just after midnight in the license manager's time zone. They reflect the prior day's usage. I believe this event is in the 4.3.5 release as well, but I don't have an instance of that to verify.

dewald13
Path Finder

In the search app, go to Status > Index activity > Indexing volume.

From there you can select your time frame, indexers to search (all), and what to split it up by. I always do it for the last 24hrs looking at any server and split by index. Like it was stated above, dont include your "_*" indexes or your summary indexes.

gowen
Path Finder

I've ended up with a suggestion from Tim Cashin, excellent Splunk sales rep. In the "Deployment Monitor" app, under "License Report" there is a table "Daily License Usage for Last 60 Days". This appears to be reasonably close.

I say "reasonably close" because you can view "By Indexer" or "By License Pool" and the two don't agree. Yesterday my two indexers were at 2.33 and 5.74 GB, but my license pool logged 13.81 GB. I have a 10 GB license and had no violation yesterday. License pool seems off by roughly x2, but only roughly (add up indexers to get 14.07, != 13.81).

It has been close enough to demonstrate the pattern of usage to my management, which was my goal.

araitz
Splunk Employee
Splunk Employee

License usage can only reliably calculated via license_usage.log, and there only by sourcetype. We are working on making this more clear in the next version of the app. The primary reason for this is that metrics.log and non-sourcetype metrics in license_usage.log are often truncated to prevent an explosion of data.

amstaff
Explorer

This is what i use:

 index=_internal metrics kb series!=_* group="per_index_thruput"| eval indexed_GB = (kb / 1024 / 1024)  | timechart eval(round((sum(indexed_GB)),2)) as TotalGB fixedrange=t span=1d

You can also schedule the search and put the result in a summary index, so you can make fast searches over a longer time period.

I_am_Jeff
Communicator

I was able to get this working as a "copy and paste" while searching over the last 7 days. (my version is 4.3.4)

Seems the first and last days may be incorrect. Perhaps if I searched at midnight...

0 Karma

gowen
Path Finder

As simply cut and pasted, this returns no output for me. I will play with it and see if I can cheer it up.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...