Knowledge Management

Query about datamodel acceleration and how data is stored

koshyk
Super Champion

I was going through the documents on Datamodel Acceleration. Can you please help me in confirming if my understanding below is correct?

  1. An Addon (TA) does the Data interpretation, classification, enrichment and normalisation. Most key value pairs are extracted during search-time.
  2. A data model then abstracts/maps multiple such datasets (and brings hierarchy) during search-time . So datamodel as such does not speed-up searches, but just abstracts to make it easy for end-user.
  3. During acceleration of datamodel, key value pairs (which were search-time in Step2) are extracted and stored as indexed key-value pairs ?
  4. The fields stored in .tsidx files, are independent of TA's and any search on it, should NOT require effort on re-extracting data from _raw?
1 Solution

martin_mueller
SplunkTrust
SplunkTrust
  1. Yup.
  2. Yeah.
  3. Fields in an accelerated data model still are search-time for most purposes. For example, index-time fields cannot be added retroactively while you can add a field to a data model and use that without re-indexing... though there will be an acceleration rebuild. From the performance point of view they behave like indexed fields, and are available through tstats.
  4. I'm not quite sure where you're going for here, data model accelerations are dependent on TAs - see question one. For already-accelerated data there's no need to descend into _raw, yes. For very recent data before acceleration happened or for very old data beyond the acceleration window the datamodel-backed search will by default descend into _raw to complete the results.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust
  1. Yup.
  2. Yeah.
  3. Fields in an accelerated data model still are search-time for most purposes. For example, index-time fields cannot be added retroactively while you can add a field to a data model and use that without re-indexing... though there will be an acceleration rebuild. From the performance point of view they behave like indexed fields, and are available through tstats.
  4. I'm not quite sure where you're going for here, data model accelerations are dependent on TAs - see question one. For already-accelerated data there's no need to descend into _raw, yes. For very recent data before acceleration happened or for very old data beyond the acceleration window the datamodel-backed search will by default descend into _raw to complete the results.

martin_mueller
SplunkTrust
SplunkTrust

It's not going to apply the regexes to the raw data, the values will already have been extracted while the acceleration searches run in the background.

0 Karma

koshyk
Super Champion

thanks Martin for your help. Accepted your answer.

0 Karma

koshyk
Super Champion

@martin_mueller. Thank you

Regarding (4) : I meant to ask once the data is in high perfomance store, the extraction is independent of TAs? or during a search, will it still have to go through all TA regex etc?

0 Karma

tomasmoser
Contributor

Yes. Once data is in accelerated DM (DMA), in other words data was added into summary index of a given data model (storing in to DM summary index is called saving into so called High Performace Analytics Store (HPA) from marketing perspective) you do NOT need any TA extraction/parsing capabilities as you already filled the fields with values in some form before indexing/writing into summary index. '

Just a comment.

Storing events into a summary index means that data will get new sourcetype - "stash". This data is not count against Splunk license (its seen as internal data). However, when you save data into a summary index under some custom sourcetype (eg. sourcetyp="mysourcetype") using `collect` command then this data might in theory be parsed by TA that has parsing rules for this custom sourcetype. 

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...