Why is the tstats command not displaying all data from a data model?

[robertlynch2020]

Hi

I have set up a data model and I am reading in millions of data lines.
The issue is some data lines are not displayed by tstats or perhaps the datamodel is not taking them in?

This is the query in tstats (2,503 events)

| tstats summariesonly=true count(All_TPS_Logs.duration) AS count FROM datamodel=MLC_TPS_DEBUG WHERE (nodename=All_TPS_Logs.User_Operations host=EXCESS_WORKFLOWS_UOB) GROUPBY All_TPS_Logs.operationIdentity

Result
All_TPS_Logs.fullyQualifiedMethod count
murex.limits.utilities.cache.statistics.CacheStatisticsTimerTask#publishCacheStatistics 2503

This is the same query in a normal search

index=mlc_live sourcetype=tps host=EXCESS_WORKFLOWS_UOB (user=* OR NOT user=*) NOT overflow=true | search name = "*" |  eval fullyQualifiedMethod = name."#".operationIdentity |eval duration = endTime - startTime | stats count(duration) as Count by fullyQualifiedMethod

This is the result - We get more data! (As expected)
fullyQualifiedMethod Count
murex.limits.utilities.cache.statistics.CacheStatisticsTimerTask#publishCacheStatistics 2503
murex.risk.control.excesses.service.DefaultExcessService#closeAll 1
murex.risk.control.excesses.service.DefaultExcessService#enlist 408
murex.risk.control.excesses.service.DefaultExcessService#query 10
murex.risk.control.excesses.service.DefaultExcessService#transition 50672

*Data that both normal search and tstats are reading*

{"endTime":1474387803162,"startTime":1474387803162,"operationIdentity":"publishCacheStatistics","name":"murex.limits.utilities.cache.statistics.CacheStatisticsTimerTask","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"BATCH-640@2016-04-13--CountryEERisk-164--ProfileNettingUTable-13067","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"4096","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"Default","key":"policy"}],"id":1957}}

Example of Data not been picked up by Datamodel

{"endTime":1474384787832,"startTime":1474384787777,"operationIdentity":"closeAll","name":"murex.risk.control.excesses.service.DefaultExcessService"}

Another one

{"endTime":1474387506930,"startTime":1474387505531,"operationIdentity":"query","name":"murex.risk.control.excesses.service.DefaultExcessService"}

It looks like the large traces get picked up and the smaller ones dont...

Any help would be super 🙂 - Going crazy here.

[efika]

Hi OP - I certainly hope that you managed to resolved this issue because it seems like I am encountering now the same issue :
https://answers.splunk.com/answers/592833/accelerated-data-model-return-results-from-the-las.html

btw,isn't pivot just some kind of a wrapper to tstats /

[kmugglet]

I've seen this as well when using summariesonly=true.
Although the datamodel page showed that acceleration is 100% completed, and I was searching within the accelerated timespan, it would only show about 14 days of data (was accelerated for 3 months)

Only way to get around it was to remove summariesonly=true, which kind of defeats the object of acceleration.

In the end, I actually converted from tstats to pivot which does ad-hoc acceleration if needed.