Why does a tstats search for an accelerated data model require allow_old_summaries=t (or: tstats has no results in some apps)

[David]

Experience Seen: in an ES environment (though not tied to ES), a | tstats search for an accelerated data model returns zero (or far fewer) results but | tstats allow_old_summaries=true returns results, even for recent data.

Alternative Experience Seen: In an ES environment (though not tied to ES), running a | tstats search in one app for an accelerated data model returns results, but there are none (or far fewer) in the other environment.

[David]

This can be caused by a mismatched datamodel search. This is not currently logged through the UI, but you can verify in search.log where you will see log messages like the following:

01-05-2016 16:42:35.818 INFO  TsidxStats - Normalized search in '/opt/splunk/var/lib/splunk/defaultdb/datamodel_summary/327_813B72E7-6743-4F46-9DE6-536F78929EDD/813B72E7-6743-4F46-9DE6-536F78929EDD/DM_Splunk_SA_CIM_Malware' did not match normalized datamodel search, attempting to re-normalize

This can happen for ES environments if you have other apps installed on the same server and are running the search from one of those other apps. E.g., you have an app that defines tags (like PAN, Cisco Firewalls, etc.) but are not named with an SA-, DA-, etc. such that they are not imported by ES. The datamodel generated through ES via the CIM app will ignore those custom tags, but when you run a search from the search app, they will be present. The two searches will not match, and you will face this issue.

This can also happen for other environments where you have an app generating a datamodel where there are non-shared configurations on nested tags. If the data model includes those configs but you run the search from someplace else, that would probably surface the same behavior.

Troubleshooting and fixing this can be a bit tedious. If you have a clue to what the issue might be, it could be simpler to just experiment with disabling non-essential apps that don't match the standard ES naming scheme (SA-, DA-, TA-*, etc.).

Process:

1. Realize that something is wrong in the first place
2. Read through search.log to see what the underlying issue is (e.g., search mismatch)
3. If this is ES, the searches may be longer than the maximum log line length by default:
   • Apply appender.searchprocessAppender.maxMessageSize = 1000000 to /etc/log-searchprocess.cfg so that you get the full ES search string
   • Re-run the search
4. Look in search.log for the datamodel search (line includes "Datamodel search:") versus the bucket search (line includes "BucketSum search:")
5. Compare those two lines using a file diff utility. You can do this via Linux open source by: Copy-paste both of those into their own files Tokenize each word on to its own line, e.g., cat /tmp/datamodelsearch.txt | perl -lpe "s/ /\n/g" | tee /tmp/datamodelsearch.txt.tokenized Diff both files to figure out the search string that is causing you the issue (e.g., diff /tmp/datamodelsearch.txt /tmp/bucketsearch.txt)
6. Grep through your etc directory to find where that exists (in my case, it was sourcetype=cisco_xml, so grep -R cisco_xml apps)
7. Disable that app / config / etc
8. Re-do steps 4-7 to make sure that everything matches now.

You may need to rebuild your summaries if that change had other implications (e.g., if the datamodel was importing that app, but shouldn't have been), but may not as well.