How can I change my alerts so they do not resend once they've already been triggered?

[Hemnaath]

Hi All, We have the below query which is getting triggered everyday based on the missing UF server from the lookup table and it creates a ticket for the same. Currently this alert creates a ticket multiple times for the same forwarder. But we need open a ticket once for each server.

For example, if testsplunk1 is "missing", it should open a ticket after 7 days. On the 8th day, if it is not resolved, it currently opens another ticket. This change should make Splunk aware that it has already opened a ticket for testsplunk1 so that it doesn't open another ticket the next day.

Current search query :

| inputlookup forwarder_assets
| makemv delim=" " avg_tcp_kbps_sparkline
| eval sum_kb = if (status == "missing", "N/A", sum_kb)
| eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline)
| eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps)
| eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps)
| rename_forwarder_type(forwarder_type) ----> This is a macro (I have removed tick symbol)

| eval current_time=now()
| eval diff_time=(current_time - last_connected)
| search status=missing
| fields hostname, forwarder_type, version, os, arch, status, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps, current_time, last_connected, diff_time
| search status=missing diff_time>604800

Kindly guide me how to write/modify the query to create a ticket once per server instead of creating tickets multiple times for the same server.

[harsmarvania57]

You can achieve this via summary index but you need to provide for how many days ticket will not create.

Let's say today ticket created for host test123 and this data will store in summary index based on splunk query and same splunk query will use sub search using join command to check whether host test123 is exist in summary index or not if it's not then output will display/ingest test123 in summary index and create ticket based on your script or whatever mechanism you used to create ticket.

Now next time when this search will run again we need to provide earliest and latest time in subsearch so let's say we are checking last 7 days of summary index data whether ticket has been created for that host or not. If ticket is created in last 7 days then splunk query will not display that hostname in output and new ticket will not generate.

Based on DMC I have created "New Report" with title missing_forwarder_summary which ingest data in summary index if any new missing forwarder found and next time when same query will run it will compare latest data with last 7 days summary index data whether missing host already came in output in last 7 days and host(s) came in last 7 days it will ignore those host(s) and only display new host(s).

For testing purpose I am running below query at every 15 minutes so it will check new missing host(s) at every 15 minutes.

| inputlookup dmc_forwarder_assets
| makemv delim=" " avg_tcp_kbps_sparkline
| eval sum_kb = if (status == "missing", "N/A", sum_kb)
| eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline)
| eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps)
| eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps)
| `dmc_rename_forwarder_type(forwarder_type)`
| search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]
| eval current_time=now()
| eval diff_time=(current_time - last_connected)
| fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps, diff_time
| search status="missing" diff_time>604800 
| join type=outer hostname [ search index=summary source=missing_forwarder_summary earliest=-7d@d latest=now | fields hostname, source]
| search NOT source="missing_forwarder_summary"
| fields - source

[Hemnaath]

Hi harsmarvania thanks for your effort on this, hey i am not good in SPL language, I know little bit of SPL queries so can I test this above query in deployment manager search portal, from where this alerts are getting triggered and the lookup table/macro are present. And is there any state table command that can be used in the above query. Kindly advise me on this.

[harsmarvania57]

Hi @Hemnaath,

To test this you need to create new schedule search on Distributed Management Console server and give title as missing_forwarder_summary & use above splunk query , schedule it to run at every 15 minutes and give earliest time -15m@m and latest time now and then enable summary indexing and select summary index as "summary" and then wait for 30-45 minutes so schedule search will run 2-3 times and ingest some data in summary index and then run above query manually to check whether you are getting expected output or not.

Thanks,
Harshil

[Hemnaath]

Hi Harshil, thanks buddy but I have question hey you had mentioned the below query which i did not understand what is the purpose of it dmc_assets lookup table contains our splunk instance details not related to UF nodes, and in second query join type=outer hostname is used can you explain me please.

search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]

and
join type=outer hostname [ search index=summary source=missing_forwarder_summary earliest=-7d@d latest=now | fields hostname, source]

apologies if had asked u a simple question. I am not good in SPL.
so kindly guide me Harshil.

[harsmarvania57]

Hi @Hemnaath,

Below query will not match your splunk indexer, search head, cluster master.

search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]

And second query is matching output of this

 | inputlookup dmc_forwarder_assets
 | makemv delim=" " avg_tcp_kbps_sparkline
 | eval sum_kb = if (status == "missing", "N/A", sum_kb)
 | eval avg_tcp_kbps_sparkline = if (status == "missing", "N/A", avg_tcp_kbps_sparkline)
 | eval avg_tcp_kbps = if (status == "missing", "N/A", avg_tcp_kbps)
 | eval avg_tcp_eps = if (status == "missing", "N/A", avg_tcp_eps)
 | `dmc_rename_forwarder_type(forwarder_type)`
 | search NOT [| inputlookup dmc_assets | dedup serverName | rename serverName as hostname | fields hostname]
 | eval current_time=now()
 | eval diff_time=(current_time - last_connected)
 | fields hostname, forwarder_type, version, os, arch, status, last_connected, sum_kb, avg_tcp_kbps_sparkline, avg_tcp_kbps, avg_tcp_eps, diff_time
 | search status="missing" diff_time>604800 

with summary index data whether Universal forwarder already triggered any alert in last 7 days or not and append that result in first query output.

and below logic will ignore those hosts from output which were down in last 7 days.

 | search NOT source="missing_forwarder_summary"
 | fields - source

I hope this clears your query and I can't explain in more detail now 😛

Thanks,
Harshil

[Hemnaath]

Hi Harshil, thanks a lot but when I had executed the " | inputlookup dmc_assets" in DMC console in our environment I could see the content of the table referring the asset details related to Splunk server instances , and also I had tried to execute the index = summary and source ="missing_forwarder_summary" as it did not fetch any information,not sure why its not fetching and this is the reason I had asked you .

[harsmarvania57]

Second query will not give you any result until and unless you will create schedule search as I mentioned earlier.

[Hemnaath]

hmm thanks Harshil for guiding me in depth. hey from which part of the world you are based out of ? i am trying to understand the splunk better but unable to do so.

[harsmarvania57]

You can refer my profile and you will able to get more idea. 🙂

[Hemnaath]

Okay so you are working out of India and which part of the country ? i am also working out of south India. I am finding some tough time in understanding the splunk and working on that.

[Hemnaath]

Hi All, Can Anyone guide me on how to create query to create a ticket once per server instead of creating tickets multiple times for the same server.

thanks in advance.

[gabarrygowin]

Hi,

Under Searches, Reports and Alerts. Format your search as normal. Select "Schedule this search" > Set your periodicity > Under Alert area > select whatever values you think fit best BUT ensure you select Throttling and set period the throttle is active for.

Really it's about how noisy the search is, how often you want woke up at night and how much load you want to put on Splunk.

Hope this helps. I'm sure someone more familiar can deep dive.

[Hemnaath]

Hi Gabarrygowin, thanks for you effort on this, actually we are getting the alert everyday based on the missing UF server from the lookup table and it creates a ticket for the same.

But actually what is our exact requirement is that currently its creating ticket multiple times for the same forwarder server eg: test01, instead of having multiple tickets for same server test01, we wanted to have single ticket created once per server.

I think we can use a state table to achieve this but not sure how to write a query using the state table.

Kindly guide me on this.