Getting Data In

How to compare events "without" common fields?

mkouzou
Explorer

Hello All,

I'm trying since 3 days now to find a solution for my problem but without success.
I look around for solutions and already asked questions but I didn't find (or I missed it) any.

I have a monitoring tool that sends me the host problems in splunk in json format.

Once the alert/problem ended, the tool send me another message in splunk but with very little informations except of one field that contains a value(alert id) of the initial alert.

Here a example:

Initial alert:

{"timestamp":1662715948,"guid":468431423,"result":1,"eid":1580,"name":"test kouzou","sev":3,"h":[{"host":"toto","name":"toto"}],"team":["titi"],"tags":-[{"tag":"App","value":"System"},{"tag":"App","value":"host-up"}]}

Recovery:

{"timestamp":1662716608,"guid":604699994,"result":0,"eid":1059134005,"init_eid":1580}

The idea is, to make searches for alerts that have not been resolved yet and show them in a dashboard, if the alert has been acknowledged by the tool, I don't want to show it .I'm trying to make equivalency with the "init_eid" and "eid" field but without success.

It is a json content and I tried to do some manual extractions in order to guarantee the "normality" of the fields.I realized also that the "transactions" command is not working at all, only for one field. Do you have any ideas on how I could achieve this goal?

Thank you in advance.

Labels (4)
0 Karma
1 Solution

mkouzou
Explorer

Hello all,

Thank you for your help.

I finally found the solution by doing that and optimizing the result by adding some more filters like severity:

 

index=my_index sourcetype=alerts  (eid=* AND result="1" AND severity>2 AND error=*) OR (init_eid=* AND result="0")
|fields result,init_eid,eid,Host,severity,error
|eval Merged_eid=coalesce(init_eid,eid)
|eval resolved=if(isnull(init_eid),"No","Yes")
|stats max(Host) as Host min(_time) as _time  max(error) as Alert max(resolved) as Resolved max(severity) as Severity by Merged_eid
|search Severity>2 AND Resolved="No"

 

Have a great day.

View solution in original post

0 Karma

mkouzou
Explorer

Hello all,

Thank you for your help.

I finally found the solution by doing that and optimizing the result by adding some more filters like severity:

 

index=my_index sourcetype=alerts  (eid=* AND result="1" AND severity>2 AND error=*) OR (init_eid=* AND result="0")
|fields result,init_eid,eid,Host,severity,error
|eval Merged_eid=coalesce(init_eid,eid)
|eval resolved=if(isnull(init_eid),"No","Yes")
|stats max(Host) as Host min(_time) as _time  max(error) as Alert max(resolved) as Resolved max(severity) as Severity by Merged_eid
|search Severity>2 AND Resolved="No"

 

Have a great day.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @mkouzou,

@gcusello search gives the active/unresolved EID values, I wanted to contribute with the below search that will show you full unresolved alerts _raw.

index=your_index 
    [ search index=test sourcetype=test2 
    | eval eid=coalesce(init_eid,eid) 
    | stats latest(result) as latest_result by eid 
    | search latest_result!=0 
    | fields eid 
    | format]

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

mkouzou
Explorer

Hello scelikok,

 

Thank you for your help.

You have correctly understood my need.

However, this search is a little bit slow, especially when we run it for the last 7 days 🙂

I finally found the solution by doing that and optimizing the result by adding some more filters like severity:

 

index=my_index sourcetype=alerts  (eid=* AND result="1" AND severity>2 AND error=*) OR (init_eid=* AND result="0")
|fields result,init_eid,eid,Host,severity,error
|eval Merged_eid=coalesce(init_eid,eid)
|eval resolved=if(isnull(init_eid),"No","Yes")
|stats max(Host) as Host min(_time) as _time  max(error) as Alert max(resolved) as Resolved max(severity) as Severity by Merged_eid
|search Severity>2 AND Resolved="No"

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mkouzou,

how can you distinguish initial alert from the recovery?

I suppose that there's a string or a field different in the two events, so if they have a different value for a field (called e.g. status) like status=initial and status=recovery, you could try something like this:

index=your_index (status="initial" OR status="recovery")
| eval EID=if(status="initial",eid,init_eid)
| stats dc(status) AS status_count BY EID
| where status_count=1

Ciao.

Giuseppe

 

0 Karma

mkouzou
Explorer

hello gcusello and thank you for your quick reply,

The "result" field changes from value "1" to value zero "0) when the problem resolved.

thank you

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mkouzou,

In this case:

index=your_index (result="1" OR result="0")
| eval EID=if(result="1",eid,init_eid)
| stats dc(result) AS result_count values(result) AS result BY EID
| where result_count=1 AND result="1"

Ciao.

Giuseppe

0 Karma

mkouzou
Explorer

hello gcusello,

 

I think that your SPL is not ok.

The result shows me always the history of the alerts.

The idea is to hide the alerts that have been resolved(result=0).

We should search for an event with "result=1", take its "eid" and search for an event containing the "init_eid" which has for sure the "result=0".

I thought that I could do that with "transaction" command but it si not working(no results).

 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mkouzou,

with my search you have as results all the events where for an eid result=1 and there isn't any result=0.

If I correctly understand this is your requirement.

Ciao.

Giuseppe

0 Karma

mkouzou
Explorer

hello gcusello,


I need all the events that they never received a "result=0" message, that is to say, only the active alerts with result=1 and not those that have been acknowledged.

For your information ,the "init_eid" and "eid" are uniques codes. We have the "eid" once the alerte triggered and the "init_eid" once the alert resolved.

The "init_eid"  contains the parent's ID ("eid").

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mkouzou,

using this search:

index=your_index (result="1" OR result="0")
| eval EID=if(result="1",eid,init_eid)
| stats dc(result) AS result_count values(result) AS result BY EID
| where result_count=1 AND result="1"

you correlate events wih result=1 (eid) with result=0 (init_eid) using the EID field to exclude the ones with both the messages (the where condition).

The logis is:

  • filter only the events with result=0 or result=1,
  • correlate events wih result=1 (eid) with result=0 (init_eid) using the EID field,
  • filtering at the end the ones with both the messages (the where condition),
  • in this way only the eid with only result=1 remains as result.

If I correctly understood, this is your requirement, what's the problem?

Do you want to find something different?

Ciao.

Giuseppe

0 Karma

mkouzou
Explorer

hello gcusello,

Thank you once again for your help.

The "problem" with your search is that is not correlating both events with same "eid & initi_eid" so splunk finds always "result"1.
I want to be able even if I chose "all time" in the time picker, to show only events that are not resolved, that is to say, that the we cannot find the couple of "eid and init_eid" (init_eid is the acknowledge event).

I don't see how your search combines these two events...:(

As I said, I tried the "transaction command" but it doesn't work...Is there any perquisites (type of data, etc) in order that the command works correctly?

 

Thank you in advance.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mkouzou,

in my search the EID field contains the values for correlation:

  • eid if result=1
  • orig_eid if  result=0

In this way running a stats command By EID you group and correlate the events for each EID.

Then with the following filter, you take only the ones with result=1, because you discard the ones with both the results and (if present) the ones with result=0.

If I correctly understood this is your goal: display all the eids (contained in the EID field) with only result=1 and not also result=0.

Ciao.

Giuseppe

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...