How to get the ingest bytes for internal logs?

[fredclown]

I know how to get the ingest bytes for non-internal logs using this ...

index=_internal source="*license_usage.log" host="{license_manager}" type=Usage

Anybody know how I could do the same for the internal logs? I'm trying to figure out how many bytes per day we ingest total (internal and non-internal logs). Thanks.

[PickleRick]

If everything else fails, you can always checkpoint your raw data size from | dbinspect

[fredclown]

Oh, I think I see what you are saying. I could potentially summarize the data every day and then subtract yesterday's bytes from todays bytes ... that might work.

[PickleRick]

Yep, that was the original idea (which I forgot as quickly as  wrote about it XD).

You just have to remember to account for buckets which were present yesterday but are not present today due to expiration and rolling to frozen. So your best bet would probably be some kind of summary indexing or keeping previous state in a lookup.

[fredclown]

So, I tried this and have a couple days worth of data. Here is the SPL I used. I have it running every morning at midnight and it pulls from a window of yesterday.

| dbinspect index=* 
| stats max(rawSize) as rawSize, values(index) as index by bucketId
| eval _time=now()
| bin _time span=1d
| stats sum(rawSize) as rawSize by _time, index
| collect index="summary"

I would expect day two to be larger than day one. Alas, it is not. It was in fact -83GB different. So, I'm not sure this method will work. : (

[PickleRick]

No. I meant checkpointing as in capturing the actual raw data sizes of every bucket for _internal index at given point in time (either by means of summary index or a lookup). Then after some time you can compare raw data sizes of all the buckets which are present (you ignore the buckets which were rolled to frozen and are no longer present).

[fredclown]

Yeah, i had thought about the freezing buckets as well. I think if I ran it every day at midnight and set the time range to yesterday I shouldn't have to worry about frozen buckets. If the time range is small enough that it would never fall outside the frozenTimePeriodInSecs and so long as my search window stays the same for each run I think that would work. I could then collect that into a summary index. Then after I get a couple days worth of summaries I could start doing the delta to get the previous day's change in bytes.

[fredclown]

Unfortunately, that won't work for getting me internal index bytes ingested per day as buckets can span multiple days.

[PickleRick]

Yes, I know. It's not a precise number, just a rough approximation. Especially if you're OK with an average number - you can sum up your data from several buckets and divide by the timerange.

But if that's not precise enough, you can always just count

index=_internal 
| stats sum(eval(len(_raw))) as total

Be warned however that it might be slow

[gcusello]

Hi @fredclown,

if you want the license consuption, you can find the search running on the License Master the search at [Settings > License > License consuption > Last 60 days], or a search from the Monitoring Console [Settings > Monitoring Console > Indexing > License Usage > Historic License Usage].

Ciao.

Giuseppe

[fredclown]

No, I don't want license consumption. I already know how to get that using the above SPL. I'm looking for bytes ingested for the  internal indexes.