Splunk Dev

How splunk internally stores events and Matrix data

santosh121
Explorer

I wanted to understand how actually splunk internally stores the events and Matrix data and what is the benefits of Matrix way of storing over events ?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, storing the values as metrics will be orders of magnitude faster than storing them as events.  You will need to redesign how you store them, however, because metrics can contain a single value along with several "dimensions" that enhance the metric.  See https://docs.splunk.com/Documentation/Splunk/8.2.0/Metrics/GetStarted for more information.

There are other ways to speed up processing.  Consider an accelerated datamodel or a summary index.

Also, make sure your queries are as efficient as possible.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

santosh121
Explorer

Can anyone suggest on the above usecase please.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Perhaps you mean "metrics" rather than "matrix"?

Events and metrics are stored very differently.  Events must be general in nature so they can store any type of data whereas metrics are rather structured and intended to store numeric data.  Metrics indexes perform on the order of 2,000 times better than event indexes.

---
If this reply helps you, Karma would be appreciated.
0 Karma

santosh121
Explorer

Thanks richgalloway.

Yeah i meant that 🙂

 

I have a data set like

We get events every sec multiple entries with the columns Env, BG, APP, Interface,BusinessField1, Valuefield1,techSuccessCount, TechnicalInprogressCnt, TechnicalFailureCnt, BusinessSuccessCount,BusinessInprogressCount,BusinessfailureCount,Valuefield2Count 

Env, BG, APP, Interface,BusinessField1, Valuefield1 (These columns are string and can be  unique row when we do stats with these.

alongwith the above fields we have techSuccessCount, TechnicalInprogressCnt, TechnicalFailureCnt, BusinessSuccessCount,BusinessInprogressCount,BusinessfailureCount,Valuefield2Count (These columns are int)

Now when we do stats everytime for last 6-8 months it is too much of processing so we are performing bin span 15m _time and storing them uniquely.

will it be to store them as metrics? 

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, storing the values as metrics will be orders of magnitude faster than storing them as events.  You will need to redesign how you store them, however, because metrics can contain a single value along with several "dimensions" that enhance the metric.  See https://docs.splunk.com/Documentation/Splunk/8.2.0/Metrics/GetStarted for more information.

There are other ways to speed up processing.  Consider an accelerated datamodel or a summary index.

Also, make sure your queries are as efficient as possible.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...