How to calculate daily volume of logs ingested by index

[jaibalaraman]

Hi Team 

I tried the below search but not getting any result, 

index=aws component=Metrics group=per_index_thruput earliest=-1w@d latest=-0d@d
| timechart span=1d sum(kb) as Usage by series
| foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024, 3)]

[bowesmana]

You can use the licence ingest log data for that also

index=_internal source=/opt/splunk/var/log/splunk/license_usage.log idx=* type=Usage
| timechart span=1d sum(b) as bytes by idx
| foreach * [ eval <<FIELD>>=round(<<FIELD>> / 1024 / 1024, 3) ]

Round/Divide as needed to get the appropriate size unit

[jamie00171]

hi @jaibalaraman 

try: 

index=_internal source=*metrics.log group=per_index_thruput earliest=-1w@d latest=-0d@d
| timechart span=1d sum(kb) as Usage by series
| eval Usage = round(Usage /1024/1024, 3)

[jaibalaraman]

yes i can see the output in the column from the below search 

source=*metrics.log group=per_index_thruput earliest=-1w@d latest=-0d@d
| timechart span=1d sum(kb) as Usage by series
| eval Usage = round(Usage /1024/1024, 3)

How do i convert the column into GB  value , also when i filter last 30 days i am able to see only last 7 days instead 30 days.  How do i fix this issue 

Note - When i specify the index , like for example index=aws i am not getting any search result from the search query ??

Thanks 

[bowesmana]

After a timechart split by a field you cannot use the field name after the timechart as it no longer exists. The field names are the values of your 'series' field.

You need to use the foreach method in your initial post

The reason why index=aws returns nothing is that the data you are searching for does not exist in that index. What makes you think it did?

It is in the _internal index, so you should definitely include that

Change your earliest/latest settings to define the time period you want to search for, or use the time picker instead and remove the earliest and latest totally.

[jaibalaraman]

Yes I tried the outcome is blank 

Question - do i need to select the time frame like last 7 days or 30 days 

[yuanliu]

You have not answered fundamental questions about your dataset.  See my comment.

BTW, once you use groupby, a single aggregation function will no longer result in field name corresponding to your AS clause.  This is why operation on Usage will not do anything. (Multiple aggregation functions will result in composite field names.  Again, operation on Usage will not do anything.)

[jaibalaraman]

Sorry i really dont understand

[yuanliu]

Have you taken away filters one by one, starting from the last one?  This is the first step to diagnose.  One key question you need to answer is: Is the groupby field named "series" extracted in Splunk?  A second question, of course, is whether the aggregated field "kb" extracted?