I have created an alert for CPU usage but the ticket is once creating and other alerts are keep on updating in the same ticket.Could anyone please help me on this?
Alert Search as follows:
index="perfmon" collection=CPU counter="% Processor Time"|stats avg(Value) as CPUusage by host| eval CPUusage=round(CPUusage,0) |where CPUusage > 10 AND CPUusage < 40
I found below information on Splunk Service Now Doc. Can you please verify the same?
If you are creating an incident, note that the behavior for the Correlation ID field is slightly different in a custom alert action than it is in the commands and scripts. This variation supports the ability to update incidents using the correlation ID in subsequent custom alert actions. In a custom alert action, if you leave this value blank, the Splunk platform does not generate a random UUID, but generates a correlation ID based on the the md5 of your alert name and the app name. Ensure that you give each alert using a custom alert action a unique name across your Splunk deployment.
Can I give something as correlation ID? If so how can I differentiate each alerts correlation ID? Could you please help me with your suggestion.
Yes, you can.
"Correlation ID" will help you to manage incident individually. So try to make "Correlation ID" unique and Dynamic. For eg, if you have an alert for CPU Monitoring of multiple hosts then create an alert (in your case) CPU Utilization with "Correlation ID"="CPU Utilization: $result.host$. This will create Incident for a particular host. If 5 host triggering alert then 5 incident will generate.
Please try and let me know for any help.
Thanks a lot @Kamlesh. This helped me to create new Incidents without duplicate hosts.
Now, can I update the same ticket priority if CPU utilization is above 40?Can I have your assistance please?
And let me know if I close one of the ticket, for same host if alert triggers again it uses the same ticket?Since I am using same correlation id?
I need to open a new ticket if alert triggers again if already raised ticket resolved.
Yes, we can update the priority of Incident. I'm not sure SNOW Custom Alert will do that.
I've tried it with SNOW Custom Command. Below is command
This command will use
| snowincident --correlation_id SOME_CORRELATION_ID --priority 1 --category software --short_description "CPU Temrature is very high" --contact_type Phone
You can use streaming command also for same.
| makeresults | eval correlation_id="SOME_CORRELATION_ID", priority="1", category="software" ,short_description="CPU Temrature is very very high",contact_type=" Phone" | snowincidentstream
In your case, SNOW Custom alert will not help. But you can try to use SNOW Custom Command (
snowincidentstream) in Splunk alert to achieve this. Please let me know for any help in this.
This works like a charm.
Let me know what correlation ID I need to give for n number of hosts?If I give the below:
"Correlation ID"="CPU Utilization: .host."
Does it open closed ticket which is already created using the above correlation ID?
An alert triggered and it created a ticket,the assignee resolved the ticket.
Now another alert from same host comes,does it open the same ticket?
And may I know how can I schedule this search for every 15 mins and include CPU Utilization of each customer in description
"Correlation ID"="CPU Utilization: $result.host$
This Correlation ID will work for you for creating Incident as well as closing the Incident.
Make sure Correlation ID in both alerts will same.
--state 7 in SNOW Generating command.
| snowincident --correlation_id SOME_CORRELATION_ID --priority 1 --category software --short_description "CPU Temrature is very high" --contact_type Phone --state 7
state=7 in streaming command.
| makeresults | eval correlation_id="SOME_CORRELATION_ID", priority="1", category="software" ,short_description="CPU Temrature is very very high",contact_type=" Phone" | eval state="7" | snowincidentstream
How can I schedule this alert. The above I ran manualy and it worked but how can I include current CPU utilization value. When I include my search it thrown error like snowincident command should use before base search.
PS :- Do we need Event Management plugin in service now to update incidents?