We recently upgraded our Enterprise Security instance to v3.0 from v2.4. After the upgrade, I noticed that Correlation Searches relying on TSIDX instances (that is, searches that begin with a command of the form "| tstats count FROM datamodel=X") do not appear to be functioning properly. I can verify that the data models are building properly, and when I try to return the same data with a "| pivot" command, it works just fine. I'd like to know why this is happening, and see if we can remediate it...I know I can convert all my correlation searches to "| pivot" commands, but I'm hoping there's a better way.
If it makes any difference, we have changed the default home directory of the data model summaries; instead of living in /opt/splunk/var/lib/splunk/index-name, they live in /opt/splunk/var/lib/splunk/index-name/db. I first noticed this problem after the change, so that may have something to do with it...
Most of our correlations (along w/ report and dashboard searches) at this time use the `summariesonly` macro which defaults to "true" based on our macro definition in the SA-Utils app. This means that we will only search across accelerated data. This is really only difference between our tstats searches and pivot that I can think would be attributing to the differences you are seeing. For testing purposes, you should be able to run searches using "| tstats count from datamodel=$model$" vs. "| tstats summariesonly=true count from datamodel=$model$". If you are seeing a discrepancy between tstats searches, this is an indication that acceleration is not complete, or is having problems. You should absolutely not have to convert any correlation searches.
I would check for the usual errors in splunkd,web_service, etc. You can also run the following search for which the "info" field will give you status:
search_activity| search search_type="dm_acceleration" OR search_type="summary_directory"
So it looks like I can see results now, but the searches are taking far longer than they should with data model acceleration - is there a "quick and easy" way to diagnose and troubleshoot issues with data model acceleration?