Splunk Enterprise Security

Splunk Enterprise Security - TSIDX-dependent Correlation Searches not working after 3.0 upgrade

BenjaminWyatt
Communicator

We recently upgraded our Enterprise Security instance to v3.0 from v2.4. After the upgrade, I noticed that Correlation Searches relying on TSIDX instances (that is, searches that begin with a command of the form "| tstats count FROM datamodel=X") do not appear to be functioning properly. I can verify that the data models are building properly, and when I try to return the same data with a "| pivot" command, it works just fine. I'd like to know why this is happening, and see if we can remediate it...I know I can convert all my correlation searches to "| pivot" commands, but I'm hoping there's a better way.

If it makes any difference, we have changed the default home directory of the data model summaries; instead of living in /opt/splunk/var/lib/splunk/index-name, they live in /opt/splunk/var/lib/splunk/index-name/db. I first noticed this problem after the change, so that may have something to do with it...

0 Karma

mcronkrite
Splunk Employee
Splunk Employee

moving the bucket location means that acceleration needs to be rebuilt.

0 Karma

hazekamp
Builder

Benjamin,

Most of our correlations (along w/ report and dashboard searches) at this time use the `summariesonly` macro which defaults to "true" based on our macro definition in the SA-Utils app. This means that we will only search across accelerated data. This is really only difference between our tstats searches and pivot that I can think would be attributing to the differences you are seeing. For testing purposes, you should be able to run searches using "| tstats count from datamodel=$model$" vs. "| tstats summariesonly=true count from datamodel=$model$". If you are seeing a discrepancy between tstats searches, this is an indication that acceleration is not complete, or is having problems. You should absolutely not have to convert any correlation searches.

Thanks,
David

0 Karma

hazekamp
Builder

I would check for the usual errors in splunkd,web_service, etc. You can also run the following search for which the "info" field will give you status:

search_activity | search search_type="dm_acceleration" OR search_type="summary_directory"

0 Karma

BenjaminWyatt
Communicator

So it looks like I can see results now, but the searches are taking far longer than they should with data model acceleration - is there a "quick and easy" way to diagnose and troubleshoot issues with data model acceleration?

0 Karma
Get Updates on the Splunk Community!

Testing out the OpenTelemetry Collector With raw Data

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...