Monitoring Splunk

Info from index=_internal does not match CMC ingest volume

tawm_12
Engager

Hello, dear Splunk Community.

I am trying to extract the ingest volume from our client's search head, but I noticed that I am getting different results depending on which method I am using.

For example, if a run the following query:

index=_internal source=*license_usage.log* type="Usage" | eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) | eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval GB=round(b/1024/1024/1024, 3) | timechart sum(GB) as Volume span=1d

 

 

I get the following table:

_time Volume

2024-11-25240.489
2024-11-26727.444
2024-11-27751.526
2024-11-28777.469
2024-11-29727.366
2024-11-30724.419
2024-12-01787.632
2024-12-02587.710

 

On the other hand, when I got to Apps > CMC > License usage > Ingest, and fetch the data for "last 7 days" (same as above) I get the following table:

_time GB

2024-11-25851.012
2024-11-26877.134
2024-11-27872.973
2024-11-28949.041
2024-11-29939.627
2024-11-30835.154
2024-12-01955.316
2024-12-02963.486

 

As you can see, there is a considerable mismatch between both results. So here's where I'm at a crossroad because I don't know which one should I trust. Based on previous topics, I notice the above query has been recommended before, even in posts from 2024.

I don't know if this is related to my user not having the appropriate capabilities or whatnot, but any insights about this issue are greatly appreciated.

Cheers, everyone.

0 Karma

dural_yyz
Motivator

I have always preferred the roll over summary generated once daily.

index=_internal source=*license_usage.log* type=RolloverSummary

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/WhatSplunklogsaboutitself

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Shareperformancedata

 

0 Karma

tawm_12
Engager

This info actually matches the data from the CMC, the only issue I have is that you can't group the volume by index (although I can group by splunk_server/indexer).

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...