topic Re: Query about datamodel acceleration and how data is stored in Knowledge Management

Query about datamodel acceleration and how data is stored

koshyk — Fri, 25 Mar 2016 00:02:51 GMT

I was going through the documents on Datamodel Acceleration. Can you please help me in confirming if my understanding below is correct?

An Addon (TA) does the Data interpretation, classification, enrichment and normalisation. Most key value pairs are extracted during search-time.
A data model then abstracts/maps multiple such datasets (and brings hierarchy) during search-time . So datamodel as such does not speed-up searches, but just abstracts to make it easy for end-user.
During acceleration of datamodel, key value pairs (which were search-time in Step2) are extracted and stored as indexed key-value pairs ?
The fields stored in .tsidx files, are independent of TA's and any search on it, should NOT require effort on re-extracting data from _raw?

Re: Query about datamodel acceleration and how data is stored

martin_mueller — Sat, 26 Mar 2016 09:03:50 GMT

Yup.
Yeah.
Fields in an accelerated data model still are search-time for most purposes. For example, index-time fields cannot be added retroactively while you can add a field to a data model and use that without re-indexing... though there will be an acceleration rebuild. From the performance point of view they behave like indexed fields, and are available through tstats.
I'm not quite sure where you're going for here, data model accelerations are dependent on TAs - see question one. For already-accelerated data there's no need to descend into _raw, yes. For very recent data before acceleration happened or for very old data beyond the acceleration window the datamodel-backed search will by default descend into _raw to complete the results.

Re: Query about datamodel acceleration and how data is stored

koshyk — Sat, 26 Mar 2016 09:13:08 GMT

@martin_mueller. Thank you

Regarding (4) : I meant to ask once the data is in high perfomance store, the extraction is independent of TAs? or during a search, will it still have to go through all TA regex etc?

Re: Query about datamodel acceleration and how data is stored

martin_mueller — Sat, 26 Mar 2016 09:19:55 GMT

It's not going to apply the regexes to the raw data, the values will already have been extracted while the acceleration searches run in the background.

Re: Query about datamodel acceleration and how data is stored

koshyk — Sat, 26 Mar 2016 09:21:02 GMT

thanks Martin for your help. Accepted your answer.

Re: Query about datamodel acceleration and how data is stored

tomasmoser — Wed, 04 Nov 2020 09:54:42 GMT

Yes. Once data is in accelerated DM (DMA), in other words data was added into summary index of a given data model (storing in to DM summary index is called saving into so called High Performace Analytics Store (HPA) from marketing perspective) you do NOT need any TA extraction/parsing capabilities as you already filled the fields with values in some form before indexing/writing into summary index. '

Just a comment.

Storing events into a summary index means that data will get new sourcetype - "stash". This data is not count against Splunk license (its seen as internal data). However, when you save data into a summary index under some custom sourcetype (eg. sourcetyp="mysourcetype") using `collect` command then this data might in theory be parsed by TA that has parsing rules for this custom sourcetype.