How to calculate daily volume of logs ingested by index?

[mike_k]

HI all, I am trying to figure out the best method for determining the volume of logs ingested into my various indexes. From various community postings, I managed to put together the following search term, which uses the previous weeks results to get an average daily kb ingested for a particular index:

index=_internal component=Metrics group=per_index_thruput earliest=-1w@d latest=-0d@d | stats sum(kb) as Usage by series | eval UsageGB=round(Usage/8/1024/1024,4) | eval daily_avg_usage=round(UsageGB/7,2)

I thought that this was giving me a reasonable answer but then I started comparing it with values provided under License Usage report, with results split by index. By comparison the "per_index_thruput" for a particular index was providing a daily average of around 15GB, whereas the License Usage report provides an average for the same index of 56GB. This appears to be the case across all indexes measured.

Whilst in this instance i can probably just use the results provided by the license usage report, i'd like to figure out why the above search is returning such a different answer to the license usage report (as this may impact on other dashboard searches that i have running).

Thanks,

[gcusello]

Hi @mike_k,

your search is completely different from the License usage, not only because you use a different parameter because you make an average in a period, instead a daily distribution.

If you make a little change in your search,

index=_internal component=Metrics group=per_index_thruput earliest=-1w@d latest=-0d@d 
| timechart span=1d sum(kb) as Usage by series
| foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024, 3)]

you'll have a result similar to the license usage.

Anyway, my hint is take and use the searches in the License Usage dashbord.

Ciao.

Giuseppe

[mike_k]

Thanks for that. I do see what your saying about the differences in what is being searched.

If i try the suggested search, however i'm still getting quite different results between the suggested search and the license usage (split by index). The value given by license usage report is roughly 1/3 the other result which i find intriguing.

I'm guessing that kb is actually 'kilo-bits', whereas the license usage report is in bytes. But even if i divide the end search result by 8 to convert to bytes, it will give me a very different answer to the license usage report values.