Splunk Enterprise Security - TSIDX-dependent Correlation Searches not working after 3.0 upgrade

[BenjaminWyatt]

We recently upgraded our Enterprise Security instance to v3.0 from v2.4. After the upgrade, I noticed that Correlation Searches relying on TSIDX instances (that is, searches that begin with a command of the form "| tstats count FROM datamodel=X") do not appear to be functioning properly. I can verify that the data models are building properly, and when I try to return the same data with a "| pivot" command, it works just fine. I'd like to know why this is happening, and see if we can remediate it...I know I can convert all my correlation searches to "| pivot" commands, but I'm hoping there's a better way.

If it makes any difference, we have changed the default home directory of the data model summaries; instead of living in /opt/splunk/var/lib/splunk/index-name, they live in /opt/splunk/var/lib/splunk/index-name/db. I first noticed this problem after the change, so that may have something to do with it...

[mcronkrite]

moving the bucket location means that acceleration needs to be rebuilt.

[hazekamp]

Benjamin,

Most of our correlations (along w/ report and dashboard searches) at this time use the `summariesonly` macro which defaults to "true" based on our macro definition in the SA-Utils app. This means that we will only search across accelerated data. This is really only difference between our tstats searches and pivot that I can think would be attributing to the differences you are seeing. For testing purposes, you should be able to run searches using "| tstats count from datamodel=$model$" vs. "| tstats summariesonly=true count from datamodel=$model$". If you are seeing a discrepancy between tstats searches, this is an indication that acceleration is not complete, or is having problems. You should absolutely not have to convert any correlation searches.

Thanks,
David

[hazekamp]

I would check for the usual errors in splunkd,web_service, etc. You can also run the following search for which the "info" field will give you status:

search_activity | search search_type="dm_acceleration" OR search_type="summary_directory"

[BenjaminWyatt]

So it looks like I can see results now, but the searches are taking far longer than they should with data model acceleration - is there a "quick and easy" way to diagnose and troubleshoot issues with data model acceleration?