Monitoring Splunk

Clarification on why our license is tripping

atulmistry
Engager

we have a license for our QA environment for 500MB. We wanted to have the same functions (deployment, alerts, security) as our production implementation but not the volume, so we can test out new data inputs, reports, etc. before pushing to production.

anyway, last week the license in our QA environment exceeded its limit, so we needed to get it reset. not a big deal. but when i run the following search over a 7 day period:

index="_internal" group="per_index_thruput" | search series!="_audit" | search series!="_internal"| eval mb=kb/1000 | timechart span="24h" sum(mb)

result

_time sum(mb)

2010-09-13T00:00:00.000+00:00 113.5911238

2010-09-14T00:00:00.000+00:00 148.8340241

2010-09-15T00:00:00.000+00:00 199.217649

2010-09-16T00:00:00.000+00:00 168.5147037

2010-09-17T00:00:00.000+00:00 168.3872045

2010-09-18T00:00:00.000+00:00 126.0590347

2010-09-19T00:00:00.000+00:00 131.917827

2010-09-20T00:00:00.000+00:00 59.22617381

i don't see us exceeding our license limit of 500MB per day. I'm assuming the _internal and _audit indexes do not count against the license. if that is true, then why did our license trip? is something wrong with my search?

thanks,

Tags (1)
1 Solution

hulahoop
Splunk Employee
Splunk Employee

The per_*_thruput metrics only account for a sample of the sources/hosts/sourcetypes. For more details on these metrics, please see Josh Rodman's blog:

NOTE: the per_x_thruput categories are not complete.  By default they show the ten busiest of each type, for each sampling window.  If you have 2000 active forwarders, you cannot expect to see the majority of them in this data.  The sampling quantity can be adjusted, but it will increase the chattiness of metrics.log, and the resulting indexing load and _internal index size. ...

The more complete source of license volume is license_audit.log. The daily indexing volume is recorded once a day just after midnight for the previous 24 hours.

View solution in original post

0 Karma

hulahoop
Splunk Employee
Splunk Employee

The per_*_thruput metrics only account for a sample of the sources/hosts/sourcetypes. For more details on these metrics, please see Josh Rodman's blog:

NOTE: the per_x_thruput categories are not complete.  By default they show the ten busiest of each type, for each sampling window.  If you have 2000 active forwarders, you cannot expect to see the majority of them in this data.  The sampling quantity can be adjusted, but it will increase the chattiness of metrics.log, and the resulting indexing load and _internal index size. ...

The more complete source of license volume is license_audit.log. The daily indexing volume is recorded once a day just after midnight for the previous 24 hours.

0 Karma

pde
Path Finder

I too am confused by this, and had prepped a similar question to ask today. viz:

splunk search "index=_internal group=per_source_thruput startdaysago=7 | search series!=_audit | search series!=_internal| eval mb=kb/1024 | timechart span=24h sum(mb)"

_time                         sum(mb)
--------------------------- ----------------
2010-09-13 00:00:00.000 UTC 14809.2371215655
2010-09-14 00:00:00.000 UTC 29875.8677501412
2010-09-15 00:00:00.000 UTC 27406.2832803688
2010-09-16 00:00:00.000 UTC 24142.7509880746
2010-09-17 00:00:00.000 UTC 17106.6712883842
2010-09-18 00:00:00.000 UTC 12746.2103596422
2010-09-19 00:00:00.000 UTC 11789.9047690882
2010-09-20 00:00:00.000 UTC  7635.3662614736

splunk search "index=_internal group=per_host_thruput startdaysago=7 | search series!=_audit | search series!=_internal| eval mb=kb/1024 | timechart span=24h sum(mb)"

_time                         sum(mb)
--------------------------- ----------------
2010-09-13 00:00:00.000 UTC 21261.2567443395
2010-09-14 00:00:00.000 UTC 43388.2257661322
2010-09-15 00:00:00.000 UTC 39761.4269084810
2010-09-16 00:00:00.000 UTC 36302.4305372761
2010-09-17 00:00:00.000 UTC 21945.8210183508
2010-09-18 00:00:00.000 UTC 15710.3877687852
2010-09-19 00:00:00.000 UTC 14434.0866328664
2010-09-20 00:00:00.000 UTC  9864.3470373860

splunk search "index=_internal group=per_index_thruput startdaysago=7 | search series!=_audit | search series!=_internal| eval mb=kb/1024 | timechart span=24h sum(mb)"

_time                         sum(mb)
--------------------------- ----------------
2010-09-13 00:00:00.000 UTC 21597.3036532481
2010-09-14 00:00:00.000 UTC 43716.7592166314
2010-09-15 00:00:00.000 UTC 40024.8323074539
2010-09-16 00:00:00.000 UTC 36338.9201157089
2010-09-17 00:00:00.000 UTC 22110.1538692767
2010-09-18 00:00:00.000 UTC 15788.8380199687
2010-09-19 00:00:00.000 UTC 14401.9610850464
2010-09-20 00:00:00.000 UTC  9783.0207494820

splunk search "index=_internal source=*license_audit.log startdaysago=7 | eval mb=todaysBytesIndexed/1024/1024 | timechart span=24h sum(mb)"

_time                         sum(mb)
--------------------------- ------------
2010-09-13 00:00:00.000 UTC      
2010-09-14 00:00:00.000 UTC 43975.131533
2010-09-15 00:00:00.000 UTC 43700.125997
2010-09-16 00:00:00.000 UTC 40018.480144
2010-09-17 00:00:00.000 UTC 36322.960874
2010-09-18 00:00:00.000 UTC 22099.668832
2010-09-19 00:00:00.000 UTC 15779.367283
2010-09-20 00:00:00.000 UTC 14394.497048

How can there be so much variation among these results?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...