Search optimization question for y’all: We have an accelerated data model to try to drive improved performance for some dashboards. It is working, but... for the one *really* large class we have, it seems to take way longer than it should for an accelerated, tstats-based search. Here is one of the tstats commands. I’m eager to see if there’s something obvious I’m missing. The main issue seems to be the values(<field>) statements immediately after the FROM clause. If I remove those, the search drops from ~80 seconds to ~8 seconds.  I.e., those values(*) items are costly. But I don’t see another way to get them in to the results… because some events don’t have those fields. (I.e., putting them in by clause will make those events disappear, which is not what we need.)  How do I get a fast return of all records and their field values?  (NOTE: In other contexts, I’ve gotten around this by adding a custom field in the data model that concatenates the fields I want, then I re-extract (rex) them after tstats line. But that seems like a ridiculous workaround … and in this case, because of the number of fields I need to return, I might as well just throw _raw in to the data model… which seems ridiculous.)   | tstats summariesonly=t values(All_Activity.sessionID) values(All_Activity.personID) values(All_Activity.chapterTitle) values(All_Activity.sectionName) values(All_Activity.behaviorDetail) values(All_Activity.action) values(All_Activity.year) values(All_Activity.searchTerm) values(All_Activity.termName) values(All_Activity.videoname) FROM datamodel="etext_behavior" WHERE All_Activity.bookTitle="Design Your First Year Experience 2020 edition" All_Activity.course="LAS 101 Fall 2020" earliest=1597714329 latest=1603071129 BY _time All_Activity.risingID All_Activity.course All_Activity.bookTitle span=1s     ... View more

We had someone bring down a search head cluster member the other day; the user had inadvertently used "ndex=myindex" instead of "index=myindex".  I.e., a typo that led Splunk to try to match against all default indexes for the user. I'd like to be able to create a Workload Management rule that de-prioritizes such searches (i.e., whether the omission of index is intentional or not). Obviously, matching on "index=*" doesn't work for this scenario.  And 'NOT index' would match instances of index way more than the intended instance at the beginning of the search. (Also recognizing that index specification is not always "index=" it can also be "index IN(".) Further, many legit searches don't include 'index' -- e.g., tstats, inputlookup, pivot, rest, makeresults, etc.  Additionally, folks are using macros which hide index designations. Upshot: How can I create a Workload Management rule that would prevent the use of ndex (instead of index) from bringing down an SHC member? ... View more

Looking at Zoom log timestamps... I'm trying to figure out timestamps (and accuracy of _time). The Zoom 'add-on' scene is a little confusing: There is the "Splunk Connect for Zoom" app (https://splunkbase.splunk.com/app/4961/)  which is listed as an 'add-on', but it has no timestamp recognition config (no props.conf at all).  Looking at Splunk Add-on for RWI - Executive Dashboard (https://splunkbase.splunk.com/app/5063/) - this *does* have a props.conf and zoom-specific configurations... but... the only thing related to timestamps are some search-time field extractions. No timestamp recognition configs. The search-time extractions are date-time strings, not epoch-time values... and are not exhaustive.  (See SPL below for analysis / comparison of timestamp values -- including these extracted fields.) I'm seeing that timestamp=none is getting assigned to every event, so that means timestamp recognition is being attempted and, presumably, failing. Which suggests that the _time value (when recognition fails) is the same as _indextime. I'm also seeing 'min' latency values of ~-18000 seconds (suggesting Splunk is occasionally successfully recognizing a timestamp, but not getting the timezone right); and positive latency of ~74,000 seconds. More evidence that Splunk is occasionally recognizing a timestamp... but not accurately. Zoom timestamp / latency diagnostic My question: Given the issues we're seeing, and the variation in timestamps in events (see analysis below), what do the developers of the add-ons (or Splunk or Zoom) recommend as an approach to accuracy of _time?  See SPL to drive analysis of your events based on grouping (stats) by event_type type event: index="<yourzoomindex>" | regex _raw = "time|start|end" | eval indextime = strftime(_indextime,"%+") `comment("NOTE: timestamp=none is a result of Splunk's timestamp parsing; occurs when it can't find (parse) a timestamp. ")` | fillnull value="-" event_type type event | stats count count(payload.time_stamp) AS payload.time_stamp count(payload.object.date_time) AS object.date_time count(payload.object.start_time) AS object.start_time count(start_time) AS start_time count(payload.object.end_time) AS object.end_time count(end_time) AS end_time count(update_time) AS update_time count(payload.object.timezone) AS object.timezone count(payload.object.occurrences{}.start_time) AS occurrences.start_time count(payload.object.recurrence.end_date_time) AS recurrence.end_date_time count(payload.object.participant.join_time) AS participant.join_time count(join_time) AS join_time count(payload.object.participant.leave_time) AS participant.leave_time count(leave_time) AS leave_time count(payload.object.participant.sharing_details.date_time) AS participant.sharing_details.date_time count(payload.object.recording_file*.recording_start) AS recording_file*.recording_start count(payload.object.recording_file*.recording_end) AS recording_file*.recording_end first(_raw) AS sample_event by event_type type event ... View more

A data model acceleration is populating summary with "friendly" values from an automatic lookup (replacing a guid-like value) that I built for "privileged" users ... a lookup that I wish to retain (for dashboard purposes). But I can't figure out how to have acceleration summary not include that "friendly" lookup value. This appears to be because 'nobody' is always the one running acceleration searches. I'd like to be able to specify the user running the acceleration query for this data model to prevent the automatic lookup, but I can't see how to specify user of acceleration searches. All other acceleration config parameters appear to be in datamodels.conf. But I'm not seeing any option to specify alternative  "owner" / "run_as" account. Is it not possible to specify owner on the acceleration searches? Or is there another path to get acceleration searches to run as a particular user? ... View more