topic Re: tstats summariesonly=t gets no results on accelerated data models in Splunk Search

tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Sat, 03 Feb 2018 18:49:17 GMT

I have installed the CIM app done all of the event typing and tagging to get my data into the data models relevant to my environment. I have accelerated those data models. It's a clustered environment with six indexers and a single search head.

If I run the tstats command with the summariesonly=t, I always get no results.

I've looked in the internal logs to see if there are any errors or warnings around acceleration or the name of the data model, but all I see are the successful searches that show the execution time and amount of events discovered.

Any ideas on how to troubleshoot this? I'd prefer for my dashboards to run only of the TSIDX data rather than raw events.

Thx.

Re: tstats summariesonly=t gets no results on accelerated data models

micahkemp — Sat, 03 Feb 2018 20:15:40 GMT

Have you checked the status of the acceleration?

Settings -> Data models -> Expand arrow next to the datamodel in question

Under "Acceleration" you should see statistics relevant to the acceleration of this specific datamodel, including percent complete.

Re: tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Mon, 05 Feb 2018 19:55:44 GMT

They're stuck at "Building" with zero disk space usage.

Re: tstats summariesonly=t gets no results on accelerated data models

micahkemp — Mon, 05 Feb 2018 20:03:43 GMT

The datamodels haven't been summarized, likely due to not having matched events to summarize, so searching with summariesonly=true is expected to return zero results.

Have you tried searching the data without summariesonly=true or via datamodel <datamodel name> search to see if it seems like the datamodel actually finds any data?

Re: tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Mon, 05 Feb 2018 20:20:06 GMT

Using | datamodel bla bla search returns results. As does searching without summariesonly=true and tstats.

We enabled acceleration on these data models a while back, so something else is the issue...

Re: tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Mon, 05 Feb 2018 23:33:16 GMT

I think the issue is that the backfill value is too high and the searches are timing out before the initial acceleration.

Re: tstats summariesonly=t gets no results on accelerated data models

cmerriman — Tue, 06 Feb 2018 12:56:22 GMT

you can search an datamodel without the acceleration built out, which is why the tstats and datamodel commands bring back data. what summary range are you accelerating? are you using Parallel summarization?

Re: tstats summariesonly=t gets no results on accelerated data models

micahkemp — Tue, 06 Feb 2018 14:41:29 GMT

Can you include your search strings for both the working and non-working searches?

Re: tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Tue, 06 Feb 2018 17:25:47 GMT

Here is what I see in the logs for the Change Analysis data model:

02-06-2018 17:12:17.529 +0000 INFO  SavedSplunker - savedsearch_id="nobody;Splunk_SA_CIM;_ACCELERATE_DM_Splunk_SA_CIM_Change_Analysis_ACCELERATE_", search_type="datamodel_acceleration", user="nobody", app="Splunk_SA_CIM", savedsearch_name="_ACCELERATE_DM_Splunk_SA_CIM_Change_Analysis_ACCELERATE_", priority=highest, status=success, digest_mode=1, scheduled_time=1517937120, window_time=0, dispatch_time=1517937120, run_time=0.492, result_count=0, alert_actions="", sid="scheduler__nobody_U3BsdW5rX1NBX0NJTQ__RMD554b34bdbf03a626a_at_1517937120_10599", suppressed=0, thread_id="AlertNotifierWorker-2"

It runs every 5 minutes, but the result count is always zero.

Yet | datamodel Change_Analysis search earliest=-5m@m returns several hundred results.

I have the backfill period set to 1 day and the timeout for the acceleration search is set to 86400 seconds. Parallelization is set to the default of 3.

This search works: | tstats prestats=t count(Malware_Attacks.signature) FROM datamodel="Malware" BY _time span=1d

This does not: | tstats summariesonly =t prestats=t count(Malware_Attacks.signature) FROM datamodel="Malware" BY _time span=1d

Re: tstats summariesonly=t gets no results on accelerated data models

kamlesh_vaghela — Wed, 07 Feb 2018 06:16:07 GMT

Hi @responsys_cm,

You are not getting any data in tstats search with and without summariesonly, right?
Well I assume you did all configuration check from data model side So is it possible to validate event side configurations? Can you please check it by executing search from constraint in data model, with each level (if any). This will return error if any. If it is then it would be cause to make data model in building state(Not build state) . Verify the eventtype and tag by executing search if it is used in any constraint. I hope this will help you.

Thanks

Re: tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Wed, 07 Feb 2018 17:04:01 GMT

With summariesonly=t, I get nothing. Without summariesonly=t, I get results.

The problem seems to be that when the acceleration searches run, they find no results. And yet | datamodel XXXX search does.

Re: tstats summariesonly=t gets no results on accelerated data models

493669 — Wed, 07 Feb 2018 17:21:12 GMT

summariesonly=t flag, this tells the data model only to look at existing accelerated data (tsidx data) that has been automatically generated by the acceleration and non-summarized data will not be provided

Re: tstats summariesonly=t gets no results on accelerated data models

kamlesh_vaghela — Wed, 07 Feb 2018 18:40:18 GMT

Without summariesonly=t you get results. Well means the datamodel don't have any accelerated data as @493669 mentioned. @responsys_cm can you please let us know the acceleration period and the volume of events for that particular period? As you mentioned above, your data-model is stuck in building state So might be this information helpful to us.

Thanks

Re: tstats summariesonly=t gets no results on accelerated data models

responsys_cm — Wed, 07 Feb 2018 19:30:40 GMT

For the high volume data sources (tens of millions of daily events), I've tried setting the backfill to only be a day. For the lower volume ones with periodic data, like vulnerability scans, those are set to a week.

But even the data models that have a relatively small amount of data and a modest backfill aren't accelerating...

Re: tstats summariesonly=t gets no results on accelerated data models

graju89 — Thu, 01 Nov 2018 15:47:06 GMT

@responsys_cm I am having the same issue. Any luck for you in fixing this?

Re: tstats summariesonly=t gets no results on accelerated data models

khowson — Tue, 29 Sep 2020 23:55:24 GMT

Try removing part of the datamodel objects in the search. so try

| tstats summariesonly count from datamodel=Network_Traffic where * by All_Traffic.src

Instead of:

| tstats summariesonly count from datamodel=Network_Traffic.All_Traffic where * by All_Traffic.src

Let meknow if that work.

Re: tstats summariesonly=t gets no results on accelerated data models

woodcock — Tue, 02 Apr 2019 04:59:47 GMT

If it doesn't work with summariesonly=f then it never will with summariesonly=t. You have to ensure that you did your CIM-mapping correctly and it sounds like you did not.

Re: tstats summariesonly=t gets no results on accelerated data models

khowson — Wed, 30 Sep 2020 00:01:07 GMT

You're missing the point. That's okay.

It does work with summariesonly=f. The issue is with summariesonly=true and the path the data is contained on the indexer.

This does not work:
| tstats summariesonly=true count from datamodel=Network_Traffic.All_Traffic where * by All_Traffic.src

however this does:
| tstats summariesonly=true count from datamodel=Network_Traffic where * by All_Traffic.src