Getting Data In
Highlighted

How can I change my alerts so they do not resend once they've already been triggered?

Motivator

Hi All, We have the below query which is getting triggered everyday based on the missing UF server from the lookup table and it creates a ticket for the same. Currently this alert creates a ticket multiple times for the same forwarder. But we need open a ticket once for each server.

For example, if testsplunk1 is "missing", it should open a ticket after 7 days. On the 8th day, if it is not resolved, it currently opens another ticket. This change should make Splunk aware that it has already opened a ticket for testsplunk1 so that it doesn't open another ticket the next day.

Current search query :

| inputlookup forwarderassets
| makemv delim=" " avg
tcpkbpssparkline
| eval sumkb = if (status == "missing", "N/A", sumkb)
| eval avgtcpkbpssparkline = if (status == "missing", "N/A", avgtcpkbpssparkline)
| eval avgtcpkbps = if (status == "missing", "N/A", avgtcpkbps)
| eval avgtcpeps = if (status == "missing", "N/A", avgtcpeps)
| renameforwardertype(forwardertype) ----> This is a macro (I have removed tick symbol)

| eval current
time=now()
| eval difftime=(currenttime - lastconnected)
| search status=missing
| fields hostname, forwarder
type, version, os, arch, status, sumkb, avgtcpkbpssparkline, avgtcpkbps, avgtcpeps, currenttime, lastconnected, difftime
| search status=missing diff
time>604800

Kindly guide me how to write/modify the query to create a ticket once per server instead of creating tickets multiple times for the same server.

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

Motivator

Hi All, Can Anyone guide me on how to create query to create a ticket once per server instead of creating tickets multiple times for the same server.

thanks in advance.

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

Path Finder

Hi,

Under Searches, Reports and Alerts. Format your search as normal. Select "Schedule this search" > Set your periodicity > Under Alert area > select whatever values you think fit best BUT ensure you select Throttling and set period the throttle is active for.

Really it's about how noisy the search is, how often you want woke up at night and how much load you want to put on Splunk.

Hope this helps. I'm sure someone more familiar can deep dive.

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

Motivator

Hi Gabarrygowin, thanks for you effort on this, actually we are getting the alert everyday based on the missing UF server from the lookup table and it creates a ticket for the same.

But actually what is our exact requirement is that currently its creating ticket multiple times for the same forwarder server eg: test01, instead of having multiple tickets for same server test01, we wanted to have single ticket created once per server.

I think we can use a state table to achieve this but not sure how to write a query using the state table.

Kindly guide me on this.

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

SplunkTrust
SplunkTrust

You can achieve this via summary index but you need to provide for how many days ticket will not create.

Let's say today ticket created for host test123 and this data will store in summary index based on splunk query and same splunk query will use sub search using join command to check whether host test123 is exist in summary index or not if it's not then output will display/ingest test123 in summary index and create ticket based on your script or whatever mechanism you used to create ticket.

Now next time when this search will run again we need to provide earliest and latest time in subsearch so let's say we are checking last 7 days of summary index data whether ticket has been created for that host or not. If ticket is created in last 7 days then splunk query will not display that hostname in output and new ticket will not generate.

Based on DMC I have created "New Report" with title missingforwardersummary which ingest data in summary index if any new missing forwarder found and next time when same query will run it will compare latest data with last 7 days summary index data whether missing host already came in output in last 7 days and host(s) came in last 7 days it will ignore those host(s) and only display new host(s).

For testing purpose I am running below query at every 15 minutes so it will check new missing host(s) at every 15 minutes.

| inputlookup dmc_forwarder_assets
| makemv delim=" " avg_tcp_kbps_sparkline
| eval sum_kb = if (status == "missing", "N/A", sum_kb)
| eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline)
| eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps)
| eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps)
| `dmc_rename_forwarder_type(forwarder_type)`
| search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]
| eval current_time=now()
| eval diff_time=(current_time - last_connected)
| fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps, diff_time
| search status="missing" diff_time>604800 
| join type=outer hostname [ search index=summary source=missing_forwarder_summary earliest=-7d@d latest=now | fields hostname, source]
| search NOT source="missing_forwarder_summary"
| fields - source
0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

Motivator

Hi harsmarvania thanks for your effort on this, hey i am not good in SPL language, I know little bit of SPL queries so can I test this above query in deployment manager search portal, from where this alerts are getting triggered and the lookup table/macro are present. And is there any state table command that can be used in the above query. Kindly advise me on this.

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

SplunkTrust
SplunkTrust

Hi @Hemnaath,

To test this you need to create new schedule search on Distributed Management Console server and give title as missingforwardersummary & use above splunk query , schedule it to run at every 15 minutes and give earliest time -15m@m and latest time now and then enable summary indexing and select summary index as "summary" and then wait for 30-45 minutes so schedule search will run 2-3 times and ingest some data in summary index and then run above query manually to check whether you are getting expected output or not.

Thanks,
Harshil

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

Motivator

Hi Harshil, thanks buddy but I have question hey you had mentioned the below query which i did not understand what is the purpose of it dmc_assets lookup table contains our splunk instance details not related to UF nodes, and in second query join type=outer hostname is used can you explain me please.

search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]

and
join type=outer hostname [ search index=summary source=missingforwardersummary earliest=-7d@d latest=now | fields hostname, source]

apologies if had asked u a simple question. I am not good in SPL.
so kindly guide me Harshil.

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

SplunkTrust
SplunkTrust

Hi @Hemnaath,

Below query will not match your splunk indexer, search head, cluster master.

search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]

And second query is matching output of this

 | inputlookup dmc_forwarder_assets
 | makemv delim=" " avg_tcp_kbps_sparkline
 | eval sum_kb = if (status == "missing", "N/A", sum_kb)
 | eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline)
 | eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps)
 | eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps)
 | `dmc_rename_forwarder_type(forwarder_type)`
 | search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]
 | eval current_time=now()
 | eval diff_time=(current_time - last_connected)
 | fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps, diff_time
 | search status="missing" diff_time>604800 

with summary index data whether Universal forwarder already triggered any alert in last 7 days or not and append that result in first query output.

and below logic will ignore those hosts from output which were down in last 7 days.

 | search NOT source="missing_forwarder_summary"
 | fields - source

I hope this clears your query and I can't explain in more detail now 😛

Thanks,
Harshil

0 Karma
Highlighted

Re: How can I change my alerts so they do not resend once they've already been triggered?

Motivator

Hi Harshil, thanks a lot but when I had executed the " | inputlookup dmcassets" in DMC console in our environment I could see the content of the table referring the asset details related to Splunk server instances , and also I had tried to execute the index = summary and source ="missingforwarder_summary" as it did not fetch any information,not sure why its not fetching and this is the reason I had asked you .

0 Karma