Monitoring Splunk

Info from index=_internal does not match CMC ingest volume

tawm_12
New Member

Hello, dear Splunk Community.

I am trying to extract the ingest volume from our client's search head, but I noticed that I am getting different results depending on which method I am using.

For example, if a run the following query:

index=_internal source=*license_usage.log* type="Usage" | eval h=if(len(h)=0 OR isnull(h),"(SQUASHED)",h) | eval s=if(len(s)=0 OR isnull(s),"(SQUASHED)",s) | eval idx=if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx) | eval GB=round(b/1024/1024/1024, 3) | timechart sum(GB) as Volume span=1d

 

 

I get the following table:

_time Volume

2024-11-25240.489
2024-11-26727.444
2024-11-27751.526
2024-11-28777.469
2024-11-29727.366
2024-11-30724.419
2024-12-01787.632
2024-12-02587.710

 

On the other hand, when I got to Apps > CMC > License usage > Ingest, and fetch the data for "last 7 days" (same as above) I get the following table:

_time GB

2024-11-25851.012
2024-11-26877.134
2024-11-27872.973
2024-11-28949.041
2024-11-29939.627
2024-11-30835.154
2024-12-01955.316
2024-12-02963.486

 

As you can see, there is a considerable mismatch between both results. So here's where I'm at a crossroad because I don't know which one should I trust. Based on previous topics, I notice the above query has been recommended before, even in posts from 2024.

I don't know if this is related to my user not having the appropriate capabilities or whatnot, but any insights about this issue are greatly appreciated.

Cheers, everyone.

0 Karma

dural_yyz
Builder

I have always preferred the roll over summary generated once daily.

index=_internal source=*license_usage.log* type=RolloverSummary

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/WhatSplunklogsaboutitself

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Shareperformancedata

 

0 Karma

tawm_12
New Member

This info actually matches the data from the CMC, the only issue I have is that you can't group the volume by index (although I can group by splunk_server/indexer).

0 Karma
Get Updates on the Splunk Community!

Easily Improve Agent Saturation with the Splunk Add-on for OpenTelemetry Collector

Agent Saturation What and Whys In application performance monitoring, saturation is defined as the total load ...

Explore the Latest Educational Offerings from Splunk [January 2025 Updates]

At Splunk Education, we are committed to providing a robust learning experience for all users, regardless of ...

Developer Spotlight with Paul Stout

Welcome to our very first developer spotlight release series where we'll feature some awesome Splunk ...