Hello community, I need to set up a dashboard that tracks the status of an alert from Splunk OnCall. An alert can have 2 to 3 statuses and I would like to retrieve the _time of each step and keep it in memory for each state (to make duration calculations in particular) : I manage to retrieve the _time for each state in a dedicated field but I cannot transfer this value to the other states:   index=oncall_prod originOnCall="Prod" incidentNumber=497764 | sort _time desc | rex field=entityDisplayName "(?<Priorité>..) - (?<Titre>.*)" | eval startAlert = if(alertType == "CRITICAL", _time, "") | eval startAlert = strftime(startAlert,"%Y-%m-%d %H:%M:%S ") | eval ackAlert = if(alertType == "ACKNOWLEDGEMENT", _time, "") | eval ackAlert = strftime(ackAlert,"%Y-%m-%d %H:%M:%S ") | eval endAlert = if(alertType == "RECOVERY", _time, "") | eval endAlert = strftime(endAlert,"%Y-%m-%d %H:%M:%S ") | table _time, incidentNumber, alertType, Priorité, Titre, startAlert, ackAlert, endAlert, ticket_EV   Do you have any idea how to do this? I searched the forum but couldn't find a solution that matched my problem. Sincerely, Rajaion ... View more

Hello community, I'm having a problem with a probably stupid addition but I can't find a solution. I make a simple query which returns me an account using a field called "routingKey": However, in this example I have duplicate routingKey but with different names (example: routingdynatrace_2 and dynatrace_2 are actually the same source). This is due to a change in the way I collect my data and this has changed the name of the routingKey. The data is however not the same (the data of the routingKey "routingdynatrace_2" is not the same as "dynatrace_2") My question is: how do I add two RoutingKey after the count to get the overall total? I tried to rename the routingKey upstream but the query does not add them after renaming. If you have any ideas, I'm interested. Sincerely, Rajaion ... View more

Hello community, I am having a problem displaying a graph. I have an index that contains incidents from several monitoring tools. I need to pull up a top 10 of the most recurring alerts (that's done). However, on this top 10, I am asked for a graph to show the evolution of the number of errors in this top per week (in order to see for example when a fix has been deployed). And this is where I encounter a problem: in my query, I have my top 10 but I have an OTHER which brings together everything that is after the top 10: Here is the query that causes this graph: index=oncall_prod | search routingKey != "routingdynatrace_cluster" | dedup incidentNumber | rename entityDisplayName as Service | timechart span=1w count by Service | sort - count limit=10   I tried to use "head" or "top" to force the display of the first 10 results only but in the case of "head", it doesn't change anything, and in the case of "top", my screen remains empty. I've searched the forum and it's often these two answers that come up but in my case, it doesn't work. Do you know how to remove the OTHER to only have the first 10 results in my graph? Sincerely, Rajaion ... View more

Hello community, I'm having a problem that's probably easy to solve, but I can't figure it out. I have a query that will query an index that contains alerts from Splunk OnCall. And I count each alert source (via the associated routingkey from OnCall) and its status (Acknowledged or not). `victorops_incidents` | sort lastAlertTime desc | dedup incidentNumber | fields * | search org="*" routingKey=** pagedPolicies{}.policy.name!=0_Reroute_alertes currentPhase!=RESOLVED | eval currentPhase=case(like(currentPhase, "%UNACKED%"), "Non acquitté", like(currentPhase, "%ACKED%"), "En cours") | eval routingKey=case(like(routingKey, "%routingcontrol-m%"), "Control-M", like(routingKey, "%dyn%"), "Dynatrace", like(routingKey, "%centreon%"), "Centreon", like(routingKey, "%servicepilot%"), "ServicePilot", like(routingKey, "%p_1%"), "P1") | rename currentPhase as Etat, routingKey as Source | chart count by Etat, Source | sort - Etat I have an almost perfect table which summarizes everything but I am missing some information: I sometimes have a source which has not generated any alert so it is absent from the table (in the screen below, I have the sources "Control-M", "Dynatrace" and "ServicePilot" but I am missing "Centreon" because the latter did not have any incidents in the period of time) : My question is the following: how to make all the sources appear but display 0 when they have not had any alerts? Best regards, Rajaion ... View more

Hello community, I'm encountering a problem that's probably simple to correct, but no matter how hard I try, I can't do it. I have a query that returns several results that I count according to the time range. This allows me to provide a graph showing the hourly load. However, I noticed that when there was no result over a time range (for example between 3:00 a.m. and 4:00 a.m.), the graph does not appear in full, I am missing the time range in question : Here is my current query: index="oncall_hp" currentPhase=UNACKED routingKey=*event* entityDisplayName!=*Local-Self-Monitoring* | dedup incidentNumber | eval Heure = strftime(_time, "%H") | stats count by Heure | rename count AS Events | sort Heure I tried to force the appearance of a "0" value if there was nothing but that didn't change: index="oncall_hp" currentPhase=UNACKED routingKey=*event* entityDisplayName!=*Local-Self-Monitoring* | dedup incidentNumber | eval Heure = strftime(_time, "%H") | stats count by Heure | rename count AS Events | eval Events=if(isnull(Events) OR len(Events)==0, "0", Events) | sort Heure   I looked on the forum to see if other people had had this problem but I couldn't find it (or I didn't look well). Do you have an idea to simply add a "0" value if a time slot is empty, and that adds it to the graph? Best regards, Rajaion ... View more

Hello community, I'm having a very specific problem and I can't find a solution after several days of attempts, all of which failed. I will explain the situation to you: we have a Splunk OnCall which serves as our hypervisor and which reports the incidents of several of our monitoring tools. Our users acknowledge alerts directly to Splunk OnCall for incident support. We then pull all this data into Splunk Enterprise (via an official plugin). For several weeks, I have been trying to make the delta between the number of alerts of a type (based on its title) and the number of times this alert has been acknowledged. For this, OnCall sends the same information back to Enterprise several times but with different details: - when an alert appears on OnCall, Enterprise has the info with the status "UNACKED". - when an alert is acknowledged, it goes up with the status "ACKED" - when an alert is over, it goes up with the status "RESOLVED". So I can have up to 3 times the same information in Enterprise.   Now that the (long) scene is set, here is my problem: I manage to output the RESOLVED and ACKED alerts in the same table, in order to make a delta between the number of RESOLVED and the number of ACKED but I cannot "align" the information. I use this search :       index=oncall_prod routingKey=* | search currentPhase=RESOLVED | dedup incidentNumber | rename entityDisplayName as Service | stats count by Service | appendcols [ search index=oncall_prod routingKey=* | search currentPhase=ACKED | dedup incidentNumber | rename entityDisplayName as Service_ACKED | stats count by Service_ACKED | rename count AS ACKED] | eval matchfield=coalesce(Service,Service_ACKED) | table Service count Service_ACKED ACKED       and the result is the following: On the screen, you can see my problem: for some alerts, there has never been an acknowledgment and suddenly, there is a shift on the lines. And when I do a delta by a simple calculation, it does it row by row so the values ​​don't mean anything because it's not comparing the right things. I tried several methods, found here and there on the forum, to properly align my table, including the following search:       index=oncall_prod routingKey=* | search currentPhase=RESOLVED | dedup incidentNumber | rename entityDisplayName as Service | stats count by Service | eval matchfield=Service | join matchfield [ search index=oncall_prod routingKey=* | search currentPhase=ACKED | dedup incidentNumber | rename entityDisplayName as Service_ACKED | stats count by Service_ACKED | rename count AS ACKED | eval matchfield=Service_ACKED] | table Service count Service_ACKED ACKED       but I can't because the result shows me ONLY the lines with both a RESOLVED and ACKED status, leaving the alerts that only had the RESOLVED status undisplayed: How to make the acknowledgments face the correct RESOLVED lines? And how to leave the rows without acknowledgments empty, with a value of 0? If you have an idea, I'm interested. Best regards, Rajaion   ... View more

Hello community, I am having a problem with a dashboard that I am setting up based on Splunk OnCall data, in order to see the acknowledgment and resolution times for alerts. In order to see the resolution period of my alerts, I made a dashboard that shows me the right information: However, I sometimes have lines with two users displayed, and no more dates: Looking at the alert in detail, I see that the item I retrieve contains two pieces of information: One for the user who acknowledged the alert, and one for the resolution, always done by the "SYSTEM" user: In the construction of my research, I cannot "impose" to keep only the "SYSTEM" user when I display the resolved alerts (in the context of acknowledged alerts, it is simpler because I filter the states ACKED upstream):       index="oncall_prod" routingKey=* | search currentPhase=RESOLVED | dedup incidentNumber | rename transitions{}.at as ack, transitions{}.by as Utilisateur, incidentNumber as N_Incident, entityDisplayName as Nom_Incident | eval create_time = strptime(startTime,"%Y-%m-%dT%H:%M:%SZ") | eval ack_time = strptime(ack,"%Y-%m-%dT%H:%M:%SZ") | eval temps_ack = tostring((ack_time - create_time), "duration") | eval create_time=((create_time)+7200) | eval ack_time=((ack_time)+7200) | eval Debut_Incident = strftime(create_time,"%Y-%m-%d %H:%M:%S ") | eval Traitement = strftime(ack_time,"%Y-%m-%d %H:%M:%S ") | eval temps_ack = strftime(strptime(temps_ack, "%H:%M:%S"), "%H:%M:%S ") | rename temps_ack as Temps_Traitement | table N_Incident, Nom_Incident, Debut_Incident, Traitement, Temps_Traitement, Utilisateur       Do you have any idea what changes I need to make to successfully see only the user linked to the resolution? I'm sure it's a stupid thing but I can't quite put my finger on it. Best regards, Rajaion ... View more

Hello community, I have a problem with a search that does not return a result. For the purposes of a dashboard, I need one of my searches, when it does not return a result, to display 0. I have already succeeded in this modification in some somewhat complex searches but for a fairly simple search, I cannot do it. Here is the example in question: Note that when I have a result, it is displayed well, my search runs correctly. I attempted to use the command "| eval ACKED = if(isnull(ACKED) OR len(ACKED)==0, "0", ACKED)" but search doesn't seem to read it:   I found several topics on similar subjects (with the use of fillnull for example) but without result :   I think it's not complicated but I can't put my finger on what's the problem, do you have any idea? Best regards, Rajaion ... View more