topic Re: How to calculate daily volume of logs ingested by index in Splunk Search

How to calculate daily volume of logs ingested by index

jaibalaraman — Wed, 07 Feb 2024 19:40:41 GMT

Hi Team

I tried the below search but not getting any result,

index=aws component=Metrics group=per_index_thruput earliest=-1w@d latest=-0d@d
| timechart span=1d sum(kb) as Usage by series
| foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024, 3)]

Re: How to calculate daily volume of logs ingested by index

yuanliu — Wed, 07 Feb 2024 21:28:51 GMT

Have you taken away filters one by one, starting from the last one? This is the first step to diagnose. One key question you need to answer is: Is the groupby field named "series" extracted in Splunk? A second question, of course, is whether the aggregated field "kb" extracted?

Re: How to calculate daily volume of logs ingested by index

jamie00171 — Wed, 07 Feb 2024 21:41:57 GMT

hi @jaibalaraman

try:

index=_internal source=*metrics.log group=per_index_thruput earliest=-1w@d latest=-0d@d
| timechart span=1d sum(kb) as Usage by series
| eval Usage = round(Usage /1024/1024, 3)

Re: How to calculate daily volume of logs ingested by index

jaibalaraman — Wed, 07 Feb 2024 21:59:25 GMT

Yes I tried the outcome is blank

Question - do i need to select the time frame like last 7 days or 30 days

Re: How to calculate daily volume of logs ingested by index

yuanliu — Wed, 07 Feb 2024 22:42:31 GMT

You have not answered fundamental questions about your dataset. See my comment.

BTW, once you use groupby, a single aggregation function will no longer result in field name corresponding to your AS clause. This is why operation on Usage will not do anything. (Multiple aggregation functions will result in composite field names. Again, operation on Usage will not do anything.)

Re: How to calculate daily volume of logs ingested by index

jaibalaraman — Wed, 07 Feb 2024 22:50:07 GMT

Sorry i really dont understand

Re: How to calculate daily volume of logs ingested by index

bowesmana — Wed, 07 Feb 2024 23:04:55 GMT

You can use the licence ingest log data for that also

index=_internal source=/opt/splunk/var/log/splunk/license_usage.log idx=* type=Usage | timechart span=1d sum(b) as bytes by idx | foreach * [ eval <<FIELD>>=round(<<FIELD>> / 1024 / 1024, 3) ]

Round/Divide as needed to get the appropriate size unit

Re: How to calculate daily volume of logs ingested by index

jaibalaraman — Wed, 07 Feb 2024 23:36:15 GMT

yes i can see the output in the column from the below search

source=*metrics.log group=per_index_thruput earliest=-1w@d latest=-0d@d
| timechart span=1d sum(kb) as Usage by series
| eval Usage = round(Usage /1024/1024, 3)

How do i convert the column into GB value , also when i filter last 30 days i am able to see only last 7 days instead 30 days. How do i fix this issue

Note - When i specify the index , like for example index=aws i am not getting any search result from the search query ??

Thanks

Re: How to calculate daily volume of logs ingested by index

bowesmana — Thu, 08 Feb 2024 02:35:14 GMT

After a timechart split by a field you cannot use the field name after the timechart as it no longer exists. The field names are the values of your 'series' field.

You need to use the foreach method in your initial post

The reason why index=aws returns nothing is that the data you are searching for does not exist in that index. What makes you think it did?

It is in the _internal index, so you should definitely include that

Change your earliest/latest settings to define the time period you want to search for, or use the time picker instead and remove the earliest and latest totally.