Monitoring Splunk

Info from index=_internal does not match CMC ingest volume

tawm_12
Engager

Hello, dear Splunk Community.

I am trying to extract the ingest volume from our client's search head, but I noticed that I am getting different results depending on which method I am using.

For example, if a run the following query:

index=_internal source=*license_usage.log* type="Usage" | eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) | eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval GB=round(b/1024/1024/1024, 3) | timechart sum(GB) as Volume span=1d

 

 

I get the following table:

_time Volume

2024-11-25240.489
2024-11-26727.444
2024-11-27751.526
2024-11-28777.469
2024-11-29727.366
2024-11-30724.419
2024-12-01787.632
2024-12-02587.710

 

On the other hand, when I got to Apps > CMC > License usage > Ingest, and fetch the data for "last 7 days" (same as above) I get the following table:

_time GB

2024-11-25851.012
2024-11-26877.134
2024-11-27872.973
2024-11-28949.041
2024-11-29939.627
2024-11-30835.154
2024-12-01955.316
2024-12-02963.486

 

As you can see, there is a considerable mismatch between both results. So here's where I'm at a crossroad because I don't know which one should I trust. Based on previous topics, I notice the above query has been recommended before, even in posts from 2024.

I don't know if this is related to my user not having the appropriate capabilities or whatnot, but any insights about this issue are greatly appreciated.

Cheers, everyone.

0 Karma

dural_yyz
Motivator

I have always preferred the roll over summary generated once daily.

index=_internal source=*license_usage.log* type=RolloverSummary

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/WhatSplunklogsaboutitself

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Shareperformancedata

 

0 Karma

tawm_12
Engager

This info actually matches the data from the CMC, the only issue I have is that you can't group the volume by index (although I can group by splunk_server/indexer).

0 Karma
Get Updates on the Splunk Community!

Index This | What’s a riddle wrapped in an enigma?

September 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

BORE at .conf25

Boss Of Regular Expression (BORE) was an interactive session run again this year at .conf25 by the brilliant ...

OpenTelemetry for Legacy Apps? Yes, You Can!

This article is a follow-up to my previous article posted on the OpenTelemetry Blog, "Your Critical Legacy App ...