topic Re: Alerts are duplicating in Alerting

Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 12:19:07 GMT

Hello,

I have set a email alert.

ID is the unique identifier my source file is text file which updates after some time whenever new activity is capture, Forwarder will re read that file again, to avoid duplication of search im using dedup ID, if I don't use dedup ID in my search it will show me numbers of result which is not equal to the file.

For e.g: my file have 3 logs before after some activity 2 more logs added in file total count is 5, however splunk is showing 8 events in GUI. to avoid this im using dedup ID.

Now, the issue is my alert is on real time im getting alot duplicated results in my email. Below is my query

index=pro sourcetype=logs Remark="xyz" | dedup ID | table ID, _time, field1. field2, field3, field4

using the above query im getting correct result on GUI but numbers of alerts generate on email.

Re: Alerts are duplicating

ITWhisperer — Mon, 18 Mar 2024 14:59:45 GMT

It is not clear what settings you have on your alert searches. Do you have overlapping time periods? What is your alert trigger (that causes multiple alerts)? Please provide more detail.

Re: Alerts are duplicating

PickleRick — Mon, 18 Mar 2024 15:10:38 GMT

Also dedup is a tricky command. It returns just first occurrence of the event with given deduped field(s) _in search order_ (which doesn't have to be what you need).

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 17:52:14 GMT

I have set it to real-time monitoring and per-result, what i have identified so far is whenever splunk reads that file it giving me alert based on it.

For e.g: If there are 3 logs of Remark="xyz" and some new record added in the file with any other or same remark it gives me alerts again for those 3 logs (remark="xyz") until the file has done reading.

To avoid this im using dedup ID, my understanding was alerts are based on search query however using this query i don't have duplicated events but my alerts are duplicating.

It is very strange for me. below is my search query,

index=pro sourcetype=logs Remark="xyz"
| dedup ID
| table ID, _time. field1, field2, field3

Hope this clears.

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 17:57:07 GMT

@PickleRick I think my alerts results are not giving me results for dedup search, instead it is reading whole file again and again.

Since im using text file and it is keep getting amend by application service till EOD. So splunk is reading file again and again till the end of day. This is why im getting duplication of events on Splunk.

Is there anyway i can avoid events duplication on universal forwarder?

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 17:57:39 GMT

@ITWhisperer

Re: Alerts are duplicating

PickleRick — Mon, 18 Mar 2024 18:13:47 GMT

OK. First things first. Don't use real-time searches (in your case real-time alerts) unless there is absolutely no other way. Real-time searches hog single CPU on a search tier and one CPU per each indexer on an indexer tier. And keep them allocated for the whole time of the search.

Secondly, if you are ingesting the same events over and over again, that's not the alerting problem, that's your onboarding done wrong.

Search for a single ID over a longer period of time and see if the events are duplicated. If they are, that's one of your problems. (another - as I said before - is searching real-time).

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 18:38:11 GMT

@PickleRick Got your point, I have search for single ID and events are not duplicating if i use dedup ID, however on my alerts query i think dedup ID is not working it is giving me results from raw events. Events are duplicating the number of records im getting on that ID (without using dedup ID) are equal to my alerts.

How can i get real time alerts based on the above scenario?

Do i have to configure data on boarding? If yes, an you guide how can i avoid my events to be duplicate.

Here is a example how UF is reading that file, suppose i have 5 events after some time 4 more events generated on that txt file, so the overall count should be 9 but instead of 9 it is showing 14 here is the breakdown of it(5 events in start + 4 events added + 5 events that were before in that file). This is how my data on-boarding.

Re: Alerts are duplicating

PickleRick — Mon, 18 Mar 2024 18:42:44 GMT

No. don't use dedup. That's the whole point. Don't use dedup and see if you are finding multiple occurrences of "the same" event.

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 18:44:08 GMT

Yes im getting multiple occurrences for the same event, as i told you before how splunk is reading my text file.

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 19:00:40 GMT

@PickleRick Is there any way my alert to send unique data in the time lapse of 24 hr, Like if any event occur with the ID="ABC" it should send email alert one time after that it ignores that event.

Re: Alerts are duplicating

PickleRick — Mon, 18 Mar 2024 22:36:00 GMT

So firstly you should get your data ingestion process right. Events should not be ingested multiple times. Since we don't know where this data comes from, we can't offer much advice here. You can open another thread in the "Getting data in" section about this problem.

Re: Alerts are duplicating

mukhan1 — Mon, 18 Mar 2024 23:10:55 GMT

@PickleRick Okay thanks but i didn't find any way to avoid duplication on UF itself earlier,

I was thinking to do it other way, what if i enable Suppress results triggering the alert and set it to 24 hours, i think each unique id event will alert once within that period.

below is the query,

index=pro sourcetype=logs Remark="xyz"
| dedup ID

Re: Alerts are duplicating

ITWhisperer — Tue, 19 Mar 2024 08:18:34 GMT

You could record which events have triggered an alert and when it was triggered in a summary index or keystore/csv and remove these from the subsequent set of results is within 24 hours.