topic Re: How to show the total size of events from a source in Splunk Search

How to show the total size of events from a source?

a212830 — Wed, 28 Sep 2022 19:25:51 GMT

Hi,

I need to show a customer that Splunk is processing their entire file, and thought a good way of doing it was to calculate the total size of events from particular sources and then comparing it to the logfile itself. Is this possible? If so, how?

Re: How to show the total size of events from a source

richgalloway — Thu, 24 Mar 2016 12:38:33 GMT

If comparing byte or character counts be aware that Splunk does not index LINE_BREAKER characters ([\r\n], by default) so allow for that in your comparison.
I would probably compare the event count in Splunk to the number of lines in the log file, assuming a 1:1 ratio. This may not work if you merge multiple lines into a single event or split lines into multiple events.

Re: How to show the total size of events from a source

woodcock — Thu, 24 Mar 2016 15:44:10 GMT

If you are suing the default LINE_BREAKER which means each line is a single event then you can count lines. If you are sending all of the data (not diverting any to nullQueue) then you can count bytes.

Both like this:

index=* source=MyFile | eval bytes=len(_raw) | stats count AS Lines sum(bytes) AS Bytes by source

Re: How to show the total size of events from a source

sloshburch — Tue, 29 Sep 2020 09:14:34 GMT

Don't forget the license_usage.log file. Assuming there is no congestion, the license_usage.log file would show for any source (s), sourcetype (st), index (i), or host (h), the bytes (b) of that event. Therefore you could add up (sum) the total bytes per that file to show the true size. Or the roll over events each night will show a summary statistic of the same.

If there is no value for those fields then you may be on an old version of splunk OR there was index congestion.

Re: How to show the total size of events from a source

deepak_acalvio — Tue, 24 Jan 2017 05:33:11 GMT

You can use license_usage.log file as suggested by SloshBurch.

here is the query:
index=_internal source="*license_usage.log*" type=Usage | stats sum(eval(b/1024/1024)) AS volume_b by s

This will give you size of each source in MBs.

Re: How to show the total size of events from a source

woodcock — Tue, 24 Jan 2017 16:36:59 GMT

You should pick the best answer that got you to a solution and click Accept to close the question.

Re: How to show the total size of events from a source

sloshburch — Wed, 25 Jan 2017 13:40:04 GMT

I used to do it this way but recently learned that this won't be 100% accurate because

it assumes that len and the license counter measure the same (they don't, len measures characters while the license counter measures bytes)
it assumes that there's no delay or lag in indexing. _indextime is not the same as _time. Sometimes forwarders get backed up and an item may be indexed some time after what it's _time value is.

Re: How to show the total size of events from a source

joesrepsol — Wed, 17 May 2017 20:09:43 GMT

GREAT query. Using this one now and very helpful. Thanks so much!

Re: How to show the total size of events from a source

woodcock — Thu, 18 May 2017 02:34:47 GMT

These are both excellent points and my answer was very US-centric and not fully qualified.

Re: How to show the total size of events from a source

woodcock — Thu, 18 May 2017 02:35:04 GMT

This is another VERY excellent point.

Re: How to show the total size of events from a source

fredclown — Wed, 28 Sep 2022 19:19:24 GMT

The question is about source so unfortunately in most environments the usage.log will not be accurate. If you have a small Splunk environment it will probably work, but Splunk squashes the values of source and host to keep the event counts down for the usage.log file. It doesn't squash index or sourcetype so those would be accurate but if you are trying to use host or source and you have an environment that is not small, most likely this will be less accurate than summing up the lengths of all the _raw data.