Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Splunk Search

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Splunk Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Re: How to count number of events and length along...

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

zacksoft_wf

Contributor

10-26-2022
01:05 AM

I want to be able to able to count the number of events and the median length of events per sourcetype in Splunk ?

I'm trying to figure out the average/median size of evets for each sourcetype.

By size, I mean the charachter length of the raw events. and then multiply the count of events with the median size to get an idea of what sourcetypes contain big events , so that I can use the data for event size reduction if that is possible.

1 Solution

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

ITWhisperer

SplunkTrust

10-26-2022
03:25 AM

I might approach it like this:

Find the median size by index and source type

Determine deviation of each event as a multiple of the median

Count the number of events and the number of events where the deviation is twice the median (you could use 3 times or something else depending on how diverse your normal data is)

Determine the percentage of outlier events your source type have.

```
index=* sourcetype=*
| eval eventSize=len(_raw)
| eventstats median(eventSize) as median_eventSize by index, sourcetype
| eval deviation = eventSize / median_eventSize
| stats count count(eval(deviation>2)) as count_doubleMedian by index, sourcetype
| eval percentage_outliers = 100*count_doubleMedian/count
| sort 0 -percentage_outliers
```

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

ITWhisperer

SplunkTrust

10-26-2022
01:57 AM

```
| eval length=length(_raw)
| stats count median(length) as median by sourcetype
```

Having said that, why not sum the lengths

```
| eval length=length(_raw)
| stats count sum(length) as total by sourcetype
```

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

zacksoft_wf

Contributor

10-26-2022
02:05 AM

What I am trying to figure out is, the median length of events for a sourcetype , to see which sourcetypes have abnormally long events, and if they unnecessary can we trim it.

Hence I wanted a SPL that would give me the median lenght of events and count of events per sourcetype and possibly another field that will multiply median * event_count to tell me which sourectyped I need to look at for log volume/event size reduction analysis.

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

ITWhisperer

SplunkTrust

10-26-2022
02:14 AM

The median may not necessarily be useful.

For example, if you had ten events with lengths of 5, 10, 10, 10, 10, 10, 10, 10, 25, 100, your median would be 10, so median * count =100, but if you add them up you get 200. The mean however is 20, so mean * count = 200.

However, if you were interested in deviations from the "norm" to find which source types have outliers, median would be better as 25 is 2.5 times the median and 100 is 10 times the median, and only 1.25 and 5 times the mean.

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

zacksoft_wf

Contributor

10-26-2022
02:28 AM

This is the SPL, I am using,

any suggestion to improve it to look at the culprit sourcetypes?

index=* sourcetype=*

| eval eventSize=len(_raw)

| table eventSize sourcetype index _raw

| sort - eventSize

| stats median(eventSize) as median_eventSize, avg(eventSize) as avg_eventSize , count(_raw) as numberOfEvents by index, sourcetype

| sort median_eventSize | head 10

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

ITWhisperer

SplunkTrust

10-26-2022
03:25 AM

I might approach it like this:

Find the median size by index and source type

Determine deviation of each event as a multiple of the median

Count the number of events and the number of events where the deviation is twice the median (you could use 3 times or something else depending on how diverse your normal data is)

Determine the percentage of outlier events your source type have.

```
index=* sourcetype=*
| eval eventSize=len(_raw)
| eventstats median(eventSize) as median_eventSize by index, sourcetype
| eval deviation = eventSize / median_eventSize
| stats count count(eval(deviation>2)) as count_doubleMedian by index, sourcetype
| eval percentage_outliers = 100*count_doubleMedian/count
| sort 0 -percentage_outliers
```

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

zacksoft_wf

Contributor

10-26-2022
05:34 AM

Thank you so much @ITWhisperer

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

yuanliu

SplunkTrust

10-26-2022
01:13 AM

On the surface, this is all it takes

`| stats count median(length) by sourcetype`

provided the field length contains the length of each event. But I'm nearly certain that some information is missing.

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

zacksoft_wf

Contributor

10-26-2022
01:39 AM

| eval length = len(_raw)

| stats count median(length) as median_length by index, sourcetype

| sort median_length

I tried something like this.

Can I count the number of events per sourcetype ? the "| stats count" doesn't give me the right result.

- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

yuanliu

SplunkTrust

10-26-2022
01:44 AM

Get Updates on the Splunk Community!

Novice observability practitioners are often overly obsessed with performance. They might approach ...

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud
Today many enterprises ...

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >>
As our brave ...