How splunk internally stores events and Matrix data

[santosh121]

I wanted to understand how actually splunk internally stores the events and Matrix data and what is the benefits of Matrix way of storing over events ?

[santosh121]

Can anyone suggest on the above usecase please.

[richgalloway]

Perhaps you mean "metrics" rather than "matrix"?

Events and metrics are stored very differently.  Events must be general in nature so they can store any type of data whereas metrics are rather structured and intended to store numeric data.  Metrics indexes perform on the order of 2,000 times better than event indexes.

[santosh121]

Thanks richgalloway.

Yeah i meant that 🙂

I have a data set like

We get events every sec multiple entries with the columns Env, BG, APP, Interface,BusinessField1, Valuefield1,techSuccessCount, TechnicalInprogressCnt, TechnicalFailureCnt, BusinessSuccessCount,BusinessInprogressCount,BusinessfailureCount,Valuefield2Count 

Env, BG, APP, Interface,BusinessField1, Valuefield1 (These columns are string and can be  unique row when we do stats with these.

alongwith the above fields we have techSuccessCount, TechnicalInprogressCnt, TechnicalFailureCnt, BusinessSuccessCount,BusinessInprogressCount,BusinessfailureCount,Valuefield2Count (These columns are int)

Now when we do stats everytime for last 6-8 months it is too much of processing so we are performing bin span 15m _time and storing them uniquely.

will it be to store them as metrics? 

[richgalloway · Accepted solution]

Yes, storing the values as metrics will be orders of magnitude faster than storing them as events.  You will need to redesign how you store them, however, because metrics can contain a single value along with several "dimensions" that enhance the metric.  See https://docs.splunk.com/Documentation/Splunk/8.2.0/Metrics/GetStarted for more information.

There are other ways to speed up processing.  Consider an accelerated datamodel or a summary index.

Also, make sure your queries are as efficient as possible.