Background to this question I've developed some dashboards in Kibana (Elastic Stack 5.5). I've been asked to reproduce those dashboards in Splunk (6.6.3). Here, I'm using the term "reproduce" loosely: the Kibana and Splunk dashboards do not need to be identical. I can exploit unique features of each platform. Kibana dashboards have the following default, out-of-the-box behavior: if you marquee-select (drag your mouse over) an area of a time-based chart, the time range of the entire dashboard—the time picker, and every visualization in the dashboard—changes ("zooms in") to match that selection. To zoom out to the previous time range, you click the browser Back button. For a demonstration of that behavior, watch my YouTube video "Splunk feature request: zooming one chart zooms entire dashboard". I like that behavior. I want to reproduce it in Splunk. So do other users. I'm flexible about the method for zooming out, but I really want this "zoom one, zoom all; time picker in sync" behavior. What I've tried I've studied the "Pan and Zoom Chart Controls" example, but that doesn't update the time picker. I'm using a global time picker that defaults to "All time": <input type="time" searchWhenChanged="true"> <label>Time range</label> <default> <earliest>0</earliest> <latest></latest> </default> </input> (Perhaps using a global time picker is one of my problems, but why? I want a "global" time range; one that applies to the entire dashboard. Do I really need to use a "local" time picker, and then set earliest and latest in every search to that particular token value?) I tried adding the following to every timechart: <selection> <set token="earliest">$start$</set> <set token="latest">$end$</set> </selection> but that resulted in "invalid earliest_time" errors: the earliest and latest tokens were being set to the literal string values "$start$" and "$end$", because the start and end tokens had not yet been defined; the <selection> element was being used (the selection event was firing) before the user did anything with the chart. This issue is described in more detailed in another question. Then I tried replacing the <set> elements with <eval> (I wasn't sure from reading the documentation that this would be valid, or would work): <selection> <eval token="earliest">if($start$="$start$",$earliest$,$start$)</eval> <eval token="latest">if($end$="$end$",$latest$,$end$)</eval> </selection> (That is, if the start and end tokens have not been defined—if the selection event has been fired without the user actually selecting an area of the chart—then set earliest and latest to their current values.) That avoids the issue of setting earliest and latest to bogus values, but uncovers new problems: Clicking "Reset" in the chart that I "zoomed" does nothing. I guess I can understand that, because I've now set the global time to that zoomed selection. After zooming, selecting "All time" from the time picker, or removing earliest and latest from the URL parameters and reloading the page, produces unpredictable results. The time range can change to something that is neither all time, nor the previously zoomed time, but some other (unpredictable, to me) time range. I've tried using a "local" time picker, and referring to the resulting $form.local.earliest$ and $form.local.latest$ tokens in the <earliest> and <latest> elements of all searches, and in the <selection> elements of time charts, but that doesn't have any effect on these problems. Details of my use case These particular dashboards show log data from "high-volume" transaction processing systems; systems that process many transactions per second. I have deliberately designed the dashboards, in Kibana and Splunk, to "work" regardless of the duration specified by the time picker: the X-axis of time-based charts automatically adjusts to match the duration. The same dashboards can be used to analyze logs across a wide variety of durations. In practice, users might initially be interested in a time range of several minutes, and then zoom in to analyze progressively narrower time ranges. Summary I'm bewildered that this behavior appears to be so hard to achieve in Splunk. I feel I must be missing something, but what? Are the typical use cases for Kibana (Elastic Stack) and Splunk so different that this out-of-the-box Kibana behavior is not also desirable in Splunk? ... View more

From the Splunk docs topic "Splunk Enterprise > Developing Views and Apps for Splunk Web > About files and directories for apps and add-ons", heading "Files and directories for apps and add-ons": To allow users to make their own customizations without being clobbered by later updates of your app, you move all the files in $APP_HOME/local/ to $APP_HOME/default/. (Note to Splunk writer, if you're reading this: at the end of the paragraph containing the sentence quoted above, you might want to change "followings file" [sic] to "following files".) Okay, I did that... but now when I edit the app in Splunk Web - for example, I edit panels in the app's dashboards - Splunk Web saves my changes to the app's local directory. While that makes perfect sense to me, it also prompts this question. I think the answer is: you can't. You need to keep a "dev" version of the app where the files are in the app's local directory. Each time you want to distribute a new version of the app, you need to (as per the quoted Splunk docs, above) copy the app's files from the local directory to the default directory. That would be a minor pain, because I'd like to be able to distribute my app locally (inside my organization's intranet) using the git repo of the development version. But it seems it's not going to be that simple, unless I live with distributing all of the app's files in its local directory (but then, see "clobbered", above). Or there is some way to tell Splunk Web to update the files in the app's default directory? ... View more

I've developed an app that contains dashboards. For now, I have deliberately not limited the searches to specific indexes. Two reasons (but not the only reasons): I developed the app in Splunk Free I'd like to avoid hardcoding index names into the searches, because I don't (can't) dictate/predict what index names customers/users will want to use I've now copied that app to an instance of Splunk Enterprise (Developer Trial license), where I have defined various indexes. For testing purposes, I now want to limit the dashboards to searching specific indexes; I don't want the results to be "polluted" by other indexes that contain data of the same sourcetype in the same time range. I'm also looking ahead to the possibility of publishing the app on Splunkbase. In which case, I might include with the app one or more indexes of sample data. I'd like to define my searches in the app in a way that makes it easy to "redirect" those searches to use different indexes without requiring me (or a user who downloads the app) to edit index=myindex conditions in multiple search strings (whether in multiple saved searches, or in base searches defined in the Simple XML of the app's dashboards). One thought that occurs to me is defining tokens at the app level (that can then be used in searches), but, as far as I know, that feature is just wishful thinking. This question is similar to a question from 2011, "How do I set a default search index per app?". ... View more

The inputs.conf documentation describes a requireHeader setting for TCP inputs: requireHeader = bool Require a header be present at the beginning of every stream. This header may be used to override indexing settings. Defaults to false. Where can I find more information, preferably with examples, on specifying a header for a TCP input, and using it to override indexing settings? In particular: Which indexing settings can I override, and how? For example, can I use requireHeader = true with some other setting(s) to override which index events (in subsequent lines in the stream) get stored in? I'm already using a stanza in transforms.conf to override sourcetype per event. I haven't tried, but I think (based on the documentation I have read) that I can use the same technique to override the index per event, using DEST_KEY = _MetaData:Index . Typically, though, for my purposes, I'm more likely to want to override the index for an entire stream rather than per event, which is why a header-based override appeals to me. Will such per-event overrides take precedence over index settings overridden via whatever method(s) requireHeader = true involves? ... View more