I've got a very large dataset which has got 50 M events each month. I've currently got 3 months indexed, so approx 150M events.
Now, when I try to build up an accelerated report it still has got nearly 2M events. This is the least possible as I still need to use quite a lot of files that have unique combinations in the data.
What would be best practice to build up a search on this data? Searches have to be over all-time data.
We also have built this in ES. Here, it only takes about 1 minute to show a result over the full 3 months worth of data.
Absolutely! Search Acceleration is great for very specific searches while Datasets are more malleable and allow for a wider variety of data analysis against the data and you can even accelerate them as well!
In terms of search speed, can you confirm that you accelerated the data model after creating it? Between that and the 'tstats' command, you should see an amazing different. You would see no difference if no such acceleration was turned on.
I've build the datamodel. It shows the following and the tstats command is not working...
ACCELERATION Rebuild Update Edit
Status 100.00% Completed
Access Count 4.
Last Access: 5/12/17 12:11:53.000 PM
Size on Disk 3059.04 MB
Summary Range 0 second(s)
Updated 5/12/17 12:01:46.000 PM
"Pivot is incredibly fast up untill 113M events, the last 50M events are very slow." <- is that still the case? Does the Job Inspector tell you anything? I'd try similar searches with tstats using the summariesonly field true and false to see if you can pinpoint more of that 50m part. Also, did you mean the most recent or the earliest (relative to _time) when you mentioned "last"? And does that happen consistently or just that one time? If just that one time then it could have been load on the indexer.
I see that I received this error message. Does anybody know what this means?
searchid='SummaryDirector1494600038.3739', totalruntime=0.15, eventcount=0,
I think the SummaryDirector items are the autogenerated accelerations. Other than 'info=failed' I'm not seeing an error message. Where did this arise? If its the acceleration then I think Splunk will self-correct.
Will try that tomorrow. I'm now rebuilding the datamodel acceleration. Hoping that will fix some things..
This is the query I used:
| tstats avg(t_0_10s) FROM datamodel=ndw_acc_datamodel
I've got a datamodel called ndwaccdatamodel, wherein a dataset (root event) lives which is called ndwaccdatamodel_set1
What of this tstats command is not working - is it showing an error message or producing no results?
| tstats values FROM datamodel=ndw_acc_datamodel to validate
t_0_10s is the right field name.
According to the docs, you can use the
summariesonly flag to restrict it to only items accelerated thus far (if you desire). Also, the
prestats will return the data like if you had used the summary indexing commands - ready for other stats after with more of the original summarization details.
If it is possible, rollup the distinctives into whatever breakout timespans you need, perhaps one hourly, one daily and one monthly and put them into a summary index. Then you can pull out of that instead of the raw data.