topic Re: How splunk internally stores events and Matrix data in Splunk Dev

How splunk internally stores events and Matrix data

santosh121 — Wed, 19 May 2021 08:12:56 GMT

I wanted to understand how actually splunk internally stores the events and Matrix data and what is the benefits of Matrix way of storing over events ?

Re: How splunk internally stores events and Matrix data

richgalloway — Wed, 19 May 2021 12:58:29 GMT

Perhaps you mean "metrics" rather than "matrix"?

Events and metrics are stored very differently. Events must be general in nature so they can store any type of data whereas metrics are rather structured and intended to store numeric data. Metrics indexes perform on the order of 2,000 times better than event indexes.

Re: How splunk internally stores events and Matrix data

santosh121 — Wed, 19 May 2021 13:16:20 GMT

Thanks richgalloway.

Yeah i meant that 🙂

I have a data set like

We get events every sec multiple entries with the columns Env, BG, APP, Interface,BusinessField1, Valuefield1,techSuccessCount, TechnicalInprogressCnt, TechnicalFailureCnt, BusinessSuccessCount,BusinessInprogressCount,BusinessfailureCount,Valuefield2Count

Env, BG, APP, Interface,BusinessField1, Valuefield1 (These columns are string and can be unique row when we do stats with these.

alongwith the above fields we have techSuccessCount, TechnicalInprogressCnt, TechnicalFailureCnt, BusinessSuccessCount,BusinessInprogressCount,BusinessfailureCount,Valuefield2Count (These columns are int)

Now when we do stats everytime for last 6-8 months it is too much of processing so we are performing bin span 15m _time and storing them uniquely.

will it be to store them as metrics?

Re: How splunk internally stores events and Matrix data

santosh121 — Sat, 22 May 2021 11:33:30 GMT

Can anyone suggest on the above usecase please.

Re: How splunk internally stores events and Matrix data

richgalloway — Sun, 23 May 2021 13:59:22 GMT

Yes, storing the values as metrics will be orders of magnitude faster than storing them as events. You will need to redesign how you store them, however, because metrics can contain a single value along with several "dimensions" that enhance the metric. See https://docs.splunk.com/Documentation/Splunk/8.2.0/Metrics/GetStarted for more information.

There are other ways to speed up processing. Consider an accelerated datamodel or a summary index.

Also, make sure your queries are as efficient as possible.