topic Re: Compare events "without" common fields in Getting Data In

How to compare events "without" common fields?

mkouzou — Fri, 09 Sep 2022 17:02:20 GMT

Hello All,

I'm trying since 3 days now to find a solution for my problem but without success.
I look around for solutions and already asked questions but I didn't find (or I missed it) any.

I have a monitoring tool that sends me the host problems in splunk in json format.

Once the alert/problem ended, the tool send me another message in splunk but with very little informations except of one field that contains a value(alert id) of the initial alert.

Here a example:

Initial alert:

{"timestamp":1662715948,"guid":468431423,"result":1,"eid":1580,"name":"test kouzou","sev":3,"h":[{"host":"toto","name":"toto"}],"team":["titi"],"tags":-[{"tag":"App","value":"System"},{"tag":"App","value":"host-up"}]}

Recovery:

{"timestamp":1662716608,"guid":604699994,"result":0,"eid":1059134005,"init_eid":1580}

The idea is, to make searches for alerts that have not been resolved yet and show them in a dashboard, if the alert has been acknowledged by the tool, I don't want to show it .I'm trying to make equivalency with the "init_eid" and "eid" field but without success.

It is a json content and I tried to do some manual extractions in order to guarantee the "normality" of the fields.I realized also that the "transactions" command is not working at all, only for one field. Do you have any ideas on how I could achieve this goal?

Thank you in advance.

Re: Compare events "without" common fields

gcusello — Fri, 09 Sep 2022 12:41:04 GMT

Hi @mkouzou,

how can you distinguish initial alert from the recovery?

I suppose that there's a string or a field different in the two events, so if they have a different value for a field (called e.g. status) like status=initial and status=recovery, you could try something like this:

index=your_index (status="initial" OR status="recovery") | eval EID=if(status="initial",eid,init_eid) | stats dc(status) AS status_count BY EID | where status_count=1

Ciao.

Giuseppe

Re: Compare events "without" common fields

mkouzou — Fri, 09 Sep 2022 12:48:03 GMT

hello gcusello and thank you for your quick reply,

The "result" field changes from value "1" to value zero "0) when the problem resolved.

thank you

Re: Compare events "without" common fields

gcusello — Fri, 09 Sep 2022 12:51:22 GMT

Hi @mkouzou,

In this case:

index=your_index (result="1" OR result="0") | eval EID=if(result="1",eid,init_eid) | stats dc(result) AS result_count values(result) AS result BY EID | where result_count=1 AND result="1"

Ciao.

Giuseppe

Re: Compare events "without" common fields

mkouzou — Fri, 09 Sep 2022 13:20:47 GMT

hello gcusello,

I think that your SPL is not ok.

The result shows me always the history of the alerts.

The idea is to hide the alerts that have been resolved(result=0).

We should search for an event with "result=1", take its "eid" and search for an event containing the "init_eid" which has for sure the "result=0".

I thought that I could do that with "transaction" command but it si not working(no results).

Re: Compare events "without" common fields

gcusello — Fri, 09 Sep 2022 13:27:48 GMT

Hi @mkouzou,

with my search you have as results all the events where for an eid result=1 and there isn't any result=0.

If I correctly understand this is your requirement.

Ciao.

Giuseppe

Re: Compare events "without" common fields

mkouzou — Fri, 09 Sep 2022 13:36:18 GMT

hello gcusello,

I need all the events that they never received a "result=0" message, that is to say, only the active alerts with result=1 and not those that have been acknowledged.

For your information ,the "init_eid" and "eid" are uniques codes. We have the "eid" once the alerte triggered and the "init_eid" once the alert resolved.

The "init_eid" contains the parent's ID ("eid").

Re: Compare events "without" common fields

gcusello — Sat, 10 Sep 2022 05:28:37 GMT

Hi @mkouzou,

using this search:

index=your_index (result="1" OR result="0") | eval EID=if(result="1",eid,init_eid) | stats dc(result) AS result_count values(result) AS result BY EID | where result_count=1 AND result="1"

you correlate events wih result=1 (eid) with result=0 (init_eid) using the EID field to exclude the ones with both the messages (the where condition).

The logis is:

filter only the events with result=0 or result=1,
correlate events wih result=1 (eid) with result=0 (init_eid) using the EID field,
filtering at the end the ones with both the messages (the where condition),
in this way only the eid with only result=1 remains as result.

If I correctly understood, this is your requirement, what's the problem?

Do you want to find something different?

Ciao.

Giuseppe

Re: Compare events "without" common fields

mkouzou — Mon, 12 Sep 2022 09:04:15 GMT

hello gcusello,

Thank you once again for your help.

The "problem" with your search is that is not correlating both events with same "eid & initi_eid" so splunk finds always "result"1.
I want to be able even if I chose "all time" in the time picker, to show only events that are not resolved, that is to say, that the we cannot find the couple of "eid and init_eid" (init_eid is the acknowledge event).

I don't see how your search combines these two events...:(

As I said, I tried the "transaction command" but it doesn't work...Is there any perquisites (type of data, etc) in order that the command works correctly?

Thank you in advance.

Re: Compare events "without" common fields

gcusello — Mon, 12 Sep 2022 09:20:01 GMT

Hi @mkouzou,

in my search the EID field contains the values for correlation:

eid if result=1
orig_eid if result=0

In this way running a stats command By EID you group and correlate the events for each EID.

Then with the following filter, you take only the ones with result=1, because you discard the ones with both the results and (if present) the ones with result=0.

If I correctly understood this is your goal: display all the eids (contained in the EID field) with only result=1 and not also result=0.

Ciao.

Giuseppe

Re: How to compare events "without" common fields?

scelikok — Mon, 12 Sep 2022 10:15:44 GMT

Hi @mkouzou,

@gcusello search gives the active/unresolved EID values, I wanted to contribute with the below search that will show you full unresolved alerts _raw.

Re: How to compare events "without" common fields?

mkouzou — Tue, 13 Sep 2022 12:54:36 GMT

Hello scelikok,

Thank you for your help.

You have correctly understood my need.

However, this search is a little bit slow, especially when we run it for the last 7 days 🙂

I finally found the solution by doing that and optimizing the result by adding some more filters like severity:

index=my_index sourcetype=alerts (eid=* AND result="1" AND severity>2 AND error=*) OR (init_eid=* AND result="0") |fields result,init_eid,eid,Host,severity,error |eval Merged_eid=coalesce(init_eid,eid) |eval resolved=if(isnull(init_eid),"No","Yes") |stats max(Host) as Host min(_time) as _time max(error) as Alert max(resolved) as Resolved max(severity) as Severity by Merged_eid |search Severity>2 AND Resolved="No"

Re: How to compare events "without" common fields?

mkouzou — Tue, 13 Sep 2022 12:56:23 GMT

Hello all,

Thank you for your help.

I finally found the solution by doing that and optimizing the result by adding some more filters like severity:

Have a great day.