@Aziz94  Mean Time to Triage is supposed to measure the difference between status New and the first update, irrespective of status. (There may be a defect in the product's use of earliest() instead of min() when comparing values in the multi-valued time field. In my test environment, the triage metric has the same value as the resolution metric when all notables are closed.) Mean Time to Resolution measures the difference between status New and status Closed. Based on the searches behind those metrics, we can combine data from the Incident Management data model, the incident_review_lookup lookup, and the reviewstatuses_lookup lookup to calculate new metrics. For reference, the status values include: New (-1, 1, or null) Unassigned (0) In Progress (2) Pending (3) Resolved (4) Closed (5) You should read and understand "Restrict notable event status transitions" <https://docs.splunk.com/Documentation/ES/latest/Admin/Customizenotables#Restrict_notable_event_status_transitions> before proceeding. Your Status Configuration options, including which transitions are allowed and by whom, may invalidate the examples below. To measure the time difference between status New and status In Progress: | tstats summariesonly=true earliest(_time) as _time from datamodel=Incident_Management by "Notable_Events_Meta.rule_id" | rename "Notable_Events_Meta.*" as "*" | eval status=2 | lookup update=true incident_updates_lookup rule_id status outputnew time | search time=* | stats earliest(_time) as create_time max(time) as in_progress_time by rule_id | eval diff=in_progress_time-create_time | stats avg(diff) as mean_assignment_time | fieldformat mean_assignment_time=tostring(mean_assignment_time, "duration") To measure the time difference between status Pending and status Closed: | tstats summariesonly=true earliest(_time) as _time from datamodel=Incident_Management by "Notable_Events_Meta.rule_id" | rename "Notable_Events_Meta.*" as "*" | eval status_pending=3, status_closed=5 | lookup update=true incident_updates_lookup rule_id status as status_pending output time as pending_time | lookup update=true incident_updates_lookup rule_id status as status_closed output time as closed_time | search pending_time=* closed_time=* | stats max(pending_time) as pending_time max(closed_time) as closed_time by rule_id | eval diff=closed_time-pending_time | stats avg(diff) as mean_closure_time | fieldformat mean_closure_time=tostring(mean_closure_time, "duration") You now have two metrics, mean_assignment_time and mean_closure_time. To measure service levels, start with the values from your service level agreements, nonfunctional requirements, etc. For example: 90% of notable events must be assigned within 10 minutes 85% of notable events must be closed within 24 hours Calculate the assignment service level: | tstats summariesonly=true earliest(_time) as _time from datamodel=Incident_Management by "Notable_Events_Meta.rule_id" | rename "Notable_Events_Meta.*" as "*" | eval status=2 | lookup update=true incident_updates_lookup rule_id status outputnew time | search time=* | stats earliest(_time) as create_time max(time) as in_progress_time by rule_id | eval diff=in_progress_time-create_time ``` 10 minutes = 600 seconds ``` | stats sum(eval(if(diff<=600, 1, 0))) as assignment_service_level_met count | eval assignment_service_level=round(100*assignment_service_level_met/count, 0)."%" Calculate the closure service level: | tstats summariesonly=true earliest(_time) as _time from datamodel=Incident_Management by "Notable_Events_Meta.rule_id" | rename "Notable_Events_Meta.*" as "*" | eval status_pending=3, status_closed=5 | lookup update=true incident_updates_lookup rule_id status as status_pending outputnew time as pending_time | lookup update=true incident_updates_lookup rule_id status as status_closed outputnew time as closed_time | search pending_time=* closed_time=* | stats max(pending_time) as pending_time max(closed_time) as closed_time by rule_id | eval diff=closed_time-pending_time ``` 24 hours = 86400 seconds ``` | stats sum(eval(if(diff<=86400, 1, 0))) as closure_service_level_met count | eval closure_service_level=round(100*closure_service_level_met/count, 0)."%" You can use the searches in a dashboard, add where or search commands to compare the service levels to your agreements, etc. ... View more

@Shadolu  DB Connect can't parse the column names out of your SQL. It's a tedious rewrite, but you could try wrapping your CTE with an outer SELECT and explicitly listing the columns: SELECT COLUMN_1, COLUMN_2, COLUMN_3, ..., COLUMN_N FROM ( WITH ... ( ... ) SELECT ... FROM ... WHERE ... ) T ... View more

@cdstealer  This doesn't appear to be an issue with your configuration. Rather, it appears to be a bug in Splunk DB Connect's implementation of SLF4J logging. The INFO messages are most likely being handled by the console and written to stderr. Anything written to stderr by a child process of splunkd will be logged to splunkd.log as an ERROR message. If you have Splunk support you can report this as a defect in a new case. ... View more

@gheribhai1234 @PickleRick does make a good point about SPL not being a typical programming language. If SPL doesn't do what you need, you do have the option of writing custom commands in Python, but that's beyond the scope of this question. ... View more

@Jackiifilwhh  A quick and non-scientific approach might just count the number of distinct transId values per sessionId and alert on unexpected count values: | stats earliest(_time) as _time dc(transId) as dc_transId by sessionId | where dc_transId!=4 You can also use algorithms from Splunk Machine Learning Toolkit. For example, this search associates a set of training transId values with a sequence value by sessionId and fits the data to a random forest classifier: | sort 0 _time | streamstats count as sequence by sessionId | fit AutoPrediction transId from sequence into Jackiifilwhh_behavior_model Th sequence field is similar to your gsn field but local rather than global. You can subsequently monitor live data with the assumption that all sessions in the time range are complete. Partial sessions will generate false positives: | sort 0 _time | streamstats count as sequence by sessionId | apply Jackiifilwhh_behavior_model | where 'predicted(transId)'!=transId   ... View more

@lalitha  Doe this search return results? index="servicenow" sourcetype="snow:sc_task" sys_class_name="sc_task" | fillnull "UnAssigned" dv_assigned_to | stats latest(*) as * by dv_number | search dv_state!="Closed Complete" dv_state!="Closed Incomplete" For ServiceNow data, also recall most inputs use the sys_updated_on column for timestamp extraction. Changes to ServiceNow tables that occur in between input intervals will be missed by the input. If a task was last updated outside your search's time range, it won't be visible in your results. ... View more

@delly_fofie  You should handle that special case in whatever way makes sense for your data. My example used epoch values. If your data uses signed 32-bit epoch values, you might treat the latest value as 2147483648 (one more than the max epoch value):   <eval token="other_time_tok_latest">if($job.latestTime$!="", strptime($job.latestTime$, "%Y-%m-%dT%H:%M:%S.%3N%z"), 2147483648)</eval>   The solution you use is dependent on your source data. You may need to further modify the example. ... View more

@RockWarriorP  The user interface is not meant to be modified, and the fields provided by the alerts/fire_alerts/{name} endpoint are document here: <https://docs.splunk.com/Documentation/Splunk/8.2.6/RESTREF/RESTsearch#alerts.2Ffired_alerts.2F.7Bname.7D>: actions Any additional alert actions triggered by this alert. alert_type Indicates if the alert was historical or real-time. digest_mode   expiration_time_rendered   savedsearch_name Name of the saved search that triggered the alert. severity Indicates the severity level of an alert. Severity level ranges from Info, Low, Medium, High, and Critical. Default is Medium. Severity levels are informational in purpose and have no additional functionality. sid The search ID of the search that triggered the alert. trigger_time The time the alert was triggered. trigger_time_rendered   triggered_alerts   Have you looked at Splunk IT Essentials Work, Splunk IT Service Intelligence, or Splunk Enterprise Security as more robust solutions, depending on your use cases? You might also consider developing your own alert dashboards or using  a third-party solution like Alert Manager (not an endorsement). ... View more

@Jaylon  Can you provide more context? The search you provided isn't a functional standalone search in any version of Splunk. ... View more

@msg4sunil  Have you tried this?       timeformat="%m/%d/%Y:%H:%M:%S%Z" starttime="07/04/2021:09:48:00Z" endtime="07/04/2021:09:48:59Z"   Or this:       timeformat="%Y-%m-%dT%H:%M:%S.%3N%Z" starttime="2021-07-04T09:48:00.000Z" endtime="2021-07-04T09:48:59.999Z"   But you probably want this:     timeformat="%Y-%m-%dT%H:%M%Z" starttime="2021-07-04T09:48Z" endtime="2021-07-04T09:49Z" unless you explicitly have millisecond precision and want to use that as an upper bound. Time ranges should be read as starttime (or earliest) >= T0 and endtime (or latest) < T1. Date and time format variables are documented at <https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables>. ... View more

@lalitha  What values for dv_state to do you see after running this search? index="servicenow" sourcetype="snow:sc_task" AND sys_class_name="sc_task" | fillnull "UnAssigned" dv_assigned_to | stats latest(*) as * by dv_number | stats count by dv_state ... View more

@timsheets13  Does the "notable" index exist? ... View more

@Aziz94  Which version of Splunk Enterprise Security are you running? Splunk Enterprise Security 7.0 introduces the Executive Summary and SOC Operations dashboards and the Mean Time to Triage and Mean Time to Resolution metrics. Whether you use Splunk Enterprise Security 7.0 or not, you can download the app from Splunkbase and read the metric searches for inspiration. (It may not be appropriate to copy and paste them here.) ... View more

@Poojitha  If you're installing the RPM, the preinstall scriptlet does this: if [ -x "$SPLUNK_HOME/bin/splunk" ] ; then echo "This looks like an upgrade of an existing Splunk Server. Attempting to stop the installed Splunk Server..." $SPLUNK_HOME/bin/splunk stop fi The installer doesn't confirm whether the stop was successful. Does your Ansible playbook include a step to stop Splunk before you attempt to upgrade in place? After stopping Splunk, you should confirm 1) Splunk stopped successfully and 2) $SPLUNK_HOME/var/run/splunk/splunkd.pid was removed. You can both confirm Splunk is stopped and automatically remove the .pid file if present by running '$SPLUNK_HOME/bin/splunk status'.   ... View more

@gheribhai1234  By "records," do you mean Splunk events or JSON array elements? An answer using spath or native JSON key-value mode depends on the context. If you're okay with an approximate value for character length, you can try a regex approach. This assumes all values of aValues are strings and includes whitespace, quotes, and commas. It's not a replacement for a JSON parser.   index=myIndex | rex "(?s)aValues\\s*:\\s*\\[\\s*(?<aValues>['\"].*?['\"])\\s*\\]" | stats sum(eval(len(aValues))) as len_aValues     ... View more

@delly_fofie  We can use the first option proposed by @niketn in Running one of two searches based on time picker selection:   <form> <label>delly_fofie_time_filters</label> <search> <query>| makeresults</query> <earliest>$other_time_tok.earliest$</earliest> <latest>$other_time_tok.latest$</latest> <done> <eval token="other_time_tok_earliest">strptime($job.earliestTime$, "%Y-%m-%dT%H:%M:%S.%3N%z")</eval> <eval token="other_time_tok_latest">strptime($job.latestTime$, "%Y-%m-%dT%H:%M:%S.%3N%z")</eval> </done> </search> <fieldset submitButton="false"> <input type="time" token="_time_tok"> <label>Time</label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> <input type="time" token="other_time_tok"> <label>Other Time</label> <default> <earliest>-24h@h</earliest> <latest>now</latest> </default> </input> </fieldset> <row> <panel> <table> <search> <query>| makeresults | addinfo | eval other_time_tok_earliest=$other_time_tok_earliest$, other_time_tok_latest=$other_time_tok_latest$</query> <earliest>$_time_tok.earliest$</earliest> <latest>$_time_tok.latest$</latest> </search> </table> </panel> </row> </form>   Note that I've updated the code for Splunk Enterprise 8.2. Simple XML and Splunk Web use JavaScript to parse eval elements. The values of job.earliestTime and job.latestTime have formats that may change between Splunk versions, and strptime() format specifiers are converted internally to Moment.js specifiers. In this Simple XML example, the first time input with token _time_tok is referenced by the table search earliest and latest values. The second time input with token other_time_tok is referenced by a global search that uses Splunk's job engine to parse the values and store the earliest and latest epoch values in other_time_tok_earliest and other_time_tok_latest, respectively. You can then reference the other_time_tok_* tokens in any search or other dashboard element. I've used them with an eval command to display their values, but you can use them anywhere you'd like, including in search (implied or explicit) and where commands. ... View more