Please find my answers in bold.   Do you need an alert if there has been a problem which has not been recovered within 15 minutes in your data even if it was recovered after 16 minutes or later? If the PROBLEM alert is not RECOVERED after 15minutes, we need to trigger a script. Are you only interested in whether the last problem (without a recovery) was over 15 minutes ago? YES Can you get multiple problems (without recovery) events for the same problem? Yes, I am running this on edge nodes which are limited hosts. It could be multiple hosts as well. Does the 15 minutes start when the PROBLEM event for the latest PROBLEM first occurs? YES Does the 15 minutes start when the PROBLEM event for the latest PROBLEM last occurs? NO How far back are you looking for these events? last 30 minutes How often are you looking for these events? Every 15 minutes     Can you check below snippet as well,     ... View more

Please find my answers in BOLD Do you need an alert if there has been a problem which has not been recovered within 15 minutes in your data even if it was recovered after 16 minutes or are you just interested in whether the last problem (without a recovery) was over 15 minutes ago? YES Can you get multiple problems (without recovery) events for the same problem i.e. do you need to know when the latest (or any) problem started (and whether it was fixed within 15 minutes)? CORRECT ... View more

index=imdc_nagios_hadoop sourcetype=icinga host=* "Load_per_CPU_core" "PROBLEM" OR "RECOVERY" | fields host | search "To: <mail-addr>" | rex field=_raw "Host:(?<src_host_1>.*) - Service:(?<Service_1>.*) State:(?<State_1>.*)" | rex field=_raw "Subject: (?<Subject>.*)" | rex field=_raw "(?<Additional_Info>.*)\nTo:" | eval Service= if(isnull(Service_1),Service_2,Service_1) ,src_host= if(isnull(src_host_1),src_host_2,src_host_1) ,State= if(isnull(State_1),State_2,State_1) | eval event_type=if(match(_raw, "Subject: PROBLEM"), "PROBLEM", "RECOVERY") | lookup hostdata_lookup.csv host as src_host | table _time src_host Service State event_type cluster isvm | search cluster=*edge* AND isvm=N | sort src_host Service _time | streamstats current=f window=1 last(_time) as previous_time last(event_type) as previous_event_type by src_host Service | eval previous_time=strftime(previous_time, "%m/%d/%Y - %H:%M:%S")   Below is the output of above query,     If the CRITICAL alert is not RECOVERED after 15minutes, we need to alert. Any help is appreciated.   ... View more

Please check below snippet.     ... View more

index=imdc_nagios_hadoop sourcetype=icinga host=* "Load_per_CPU_core" "PROBLEM" | fields host | transaction host startswith="To:" | search "To: <mail-addr>" | rex field=_raw "Host:(?<src_host_1>.*) - Service:(?<Service_1>.*) State:(?<State_1>.*)" | rex field=_raw "Subject: (?<Subject>.*)" | rex field=Subject "PROBLEM - (?<src_host_2>.*) - (?<Service_2>.*) is (?<State_2>.*)" | rex field=_raw "(?<Additional_Info>.*)\nTo:" | eval Service= if(isnull(Service_1),Service_2,Service_1) ,src_host= if(isnull(src_host_1),src_host_2,src_host_1) ,State= if(isnull(State_1),State_2,State_1) | fields host ,Service,src_host,State,Subject,Additional_Info | lookup hostdata_lookup.csv host as src_host | table src_host,Service,State,_time, cluster, isvm | rename _time as Start_time | search isvm=N AND cluster=*EDGE* | eval Start_time=strftime(Start_time, "%m/%d/%Y - %H:%M:%S") | sort Start_time   index=imdc_nagios_hadoop sourcetype=icinga host=* "Load_per_CPU_core" "RECOVERY" | fields host | transaction host startswith="To:" | search "To: <mail-addr>" | rex field=_raw "Host:(?<src_host_1>.*) - Service:(?<Service_1>.*) State:(?<State_1>.*)" | rex field=_raw "Subject: (?<Subject>.*)" | rex field=Subject "RECOVERY - (?<src_host_2>.*) - (?<Service_2>.*) is (?<State_2>.*)" | rex field=_raw "(?<Additional_Info>.*)\nTo:" | eval Service= if(isnull(Service_1),Service_2,Service_1) ,src_host= if(isnull(src_host_1),src_host_2,src_host_1) ,State= if(isnull(State_1),State_2,State_1) | fields host ,Service,src_host,State,Subject,Additional_Info | lookup hostdata_lookup.csv host as src_host | table src_host,Service,State,_time, cluster, isvm | rename _time as End_time | search isvm=N AND cluster=*EDGE* | eval End_time=strftime(End_time, "%m/%d/%Y - %H:%M:%S") | sort End_time   No, recovery has events. As i said, one search will give us "Icinga Problem" and i have another search that will give us "Icinga Recovery". Using join, Icinga Problem Start time and Icinga Recovery End time, if the recovery is more than 15 minutes, need to trigger alert. ... View more

Below is the search query for icinga Problem and events too.   Below is the search query for Icinga Recovery and events.     If you want me to get rid of transaction command, thats fine. I would like to group multiple events into a single meta-event that represents a single physical event. ... View more

Please check the code, i have shared as requested. Its the same for Recovery search as well. ... View more

    index=imdc_nagios_hadoop sourcetype=icinga host=* "Load_per_CPU_core" "PROBLEM" | fields host | transaction host startswith="To:" | search "To: <Mail-Address>" | rex field=_raw "Host:(?<src_host_1>.*) - Service:(?<Service_1>.*) State:(?<State_1>.*)" | rex field=_raw "Subject: (?<Subject>.*)" | rex field=Subject "PROBLEM - (?<src_host_2>.*) - (?<Service_2>.*) is (?<State_2>.*)" | rex field=_raw "(?<Additional_Info>.*)\nTo:" | eval Service= if(isnull(Service_1),Service_2,Service_1) ,src_host= if(isnull(src_host_1),src_host_2,src_host_1) ,State= if(isnull(State_1),State_2,State_1) | fields host ,Service,src_host,State,Subject,Additional_Info | lookup hostdata_lookup.csv host as src_host | table src_host,Service,State,_time, cluster, isvm | rename _time as Start_time | search isvm=N AND cluster=*EDGE* | eval Start_time=strftime(Start_time, "%m/%d/%Y - %H:%M:%S") | sort Start_time   For security reason, removed Mail-addr   ... View more

Need to compare Host with Start_time(Icinga Problem) and End_time(Icinga Recovery), if the alert has been recovered within SLA( i.e, 15 minutes) take action or else nothing. Any help is appreciated. ... View more

No results after executing the query. There is a lookup file called "bd_users_hierarchy.csv" which contains Active Directory users and "mapr_ticket_contacts.csv " where in UseCase information exists. Please check below screenshot and query i have written to find out Top CPU Users and Usecases on all edge nodes.   In the inputlookup file called ""mapr_ticket_contacts.csv", Usecases ends with letter "s,q,g,p" need to trim down and get email addresses. For example If i remove the letter "p"   Edge Node Information  --- Edge_Nodes_All.csv Active Directory Users  --- bd_users_hierarchy.csv UseCases -- mapr_ticket_contacts.csv ( Need to trim down letter "s,q,g,p")   I have tried with the below splunk query, but not getting results index=imdc_*_os sourcetype=ps1 [|inputlookup Edge_Nodes_All.csv where Environment="*" AND host="*" |fields host] |fields cluster, host, user, total_cpu | join type=inner host [search `gold_mpstat` OR `silver_mpstat` OR `platinum_mpstat` OR `palladium_mpstat` [|inputlookup Edge_Nodes_All.csv where Environment="*" AND host="*" |fields host] |stats max(eval(id+1)) as cores by host] |eval pct_CPU = round(total_cpu/cores,2) |stats max(total_cpu) as total_cpu, max(pct_CPU) as "CPU %" by user,host,cores |table host user cores total_cpu,"CPU %" | search NOT user IN ("root","imdcsup","hadpsup") |sort - "CPU %"|head 10 | join type=left user [| inputlookup bd_users_hierarchy.csv| rename email as user_email | table user,user_email] | join type=left user [| inputlookup mapr_ticket_contacts.csv | eventstats max(Modified_Time) as Modified_Time_max by UseCase | where Modified_Time=Modified_Time_max | eval Modified_Time=if(Modified_Time=0,"Not Updated",strftime(Modified_Time,"%Y-%m-%d %H:%M")) | rename Updated_By as "Last_Updated_By",Modified_Time as "Last_Modified_Time" | rex field=UseCase "(?<UseCase>.*)."   | rename UseCase as user | rename Support_Team_DL as user_email | table user,user_email] Appreciate your quick response on the same.   ... View more

Yes ... View more

@gcusello Results are empty for App1 and App2. ... View more

@ITWhisperer @gcusello In Hadoop ResourceManager,  Once after "Operation=Submit Application Request" resourcemanager will "Allocate New ApplicationID". I would like to see how much time difference between 2 sub searches in the splunk query.    ... View more

I have already shared before, events are in HTML.   Disposition: inline Subject: INFO - Services are in Maintenance Mode over 2 hours -- AtWork-CIW-E1 Content-Type: text/html <font size=3 color=black>Hi Team,</br></br>Please find below servers which are in maintenance mode for more than 2 hours; </br></br></font> <table border=2> <TR bgcolor=#D6EAF8><TH colspan=2>Cluster Name: AtWork-CIW-E1</TH></TR> <TR bgcolor=#D6EAF8><TH colspan=1>Service</TH><TH colspan=1>Maintenance Start Time in MST</TH></TR><TR bgcolor=#FFB6C1><TH colspan=1>oozie</TH><TH colspan=1>Mon Oct 16 07:29:46 MST 2023</TH></TR> </table> <font size=3 color=black></br>  Please check in Bold characters. I want this in table format   ... View more

No ... View more

Here is my Splunk query,  Output is not good rex max_match=0 ^\w+:\s+\w+\.\w+@\w+\.\w+\s+\w+:\s+\w+\-\w+\-\w+@\w+\.\w+\s+\w+\-\w+:\s+\d+\.\d+\s+\w+\-\w+:\s+\w+\s+\w+:\s+\w+\s+\-\s+(?P<Info>\w+\s+\w+\s+\w+\s+\w+\s+\w+\s+\w+\s+\d+\s+\w+)\s+\-\-\s+(?P<ClusterName>\w+\-\w+\-\w+) |rex "(?ms)^(?:[^>\\n]*>){2}(?P<Svc>\\w+)[^=\\n]*=\\d+>(?P<Maint>[^<]+)" | table Info ClusterName Svc Maint   Info ClusterName Svc Maint Services are in Maintenance Mode over 2 hours AtWork-CIW-E1 Service Maintenance Start Time in MST     oozie Mon Oct 16 07:29:46 MST 2023   In the above output, it is capturing Service and Maintenance Start time in MST in the field extractions ... View more

MIME-Version: 1.0 Content-Disposition: inline Subject: INFO - Services are in Maintenance Mode over 2 hours -- AtWork-CIW-E1 Content-Type: text/html <font size=3 color=black>Hi Team,</br></br>Please find below servers which are in maintenance mode for more than 2 hours; </br></br></font> <table border=2> <TR bgcolor=#D6EAF8><TH colspan=2>Cluster Name: AtWork-CIW-E1</TH></TR> <TR bgcolor=#D6EAF8><TH colspan=1>Service</TH><TH colspan=1>Maintenance Start Time in MST</TH></TR> <TR bgcolor=#FFB6C1><TH colspan=1>oozie</TH><TH colspan=1>Mon Oct 16 07:29:46 MST 2023</TH></TR> </table> <font size=3 color=black></br> Script Path:/amex/ansible/maintenance_mode_service</font> <font size=3 color=black></br></br>Thank you,</br>BDP Spark Support Team</font>                                                                                                                                                     Need field extractions of the following.   Cluster Name: AtWork-CIW-E1 Service Maintenance Start Time in MST oozie Mon Oct 16 07:29:46 MST 2023   ... View more

It is not working since mvzip will process only X and Y fields. It worked with the below. eval zip= mvzip(mvzip(Cluster,Current),Max)|mvexpand zip|eval zip=split(zip, ","), Cluster=mvindex(zip, 0), Current=mvindex(zip, 1),Max=mvindex(zip, 2) I am left with threshold, if the current is above 85% than Max, it should trigger an alert.   ... View more

  How to split the above table in one line each and wanted to have threshold if the current Block size exceeds Max Block size i.,e (85%) i want to trigger alert ... View more

PI-job application_1681360813939_33163 MAPREDUCE Thu May 4 04:30:14 MST 2023 Wed Dec 31 17:00:00 MST 1969 UNDEFINED default [Thu May 04 04 Exceeded cadence2 Spark-job application_1681360813939_33167 SPARK Thu May 4 04:31:17 MST 2023 Wed Dec 31 17:00:00 MST 1969 UNDEFINED default [Thu May 04 04 Exceeded cadence2 Spark Python Pi-job application_1681360813939_33169 SPARK Thu May 4 04:31:48 MST 2023 Wed Dec 31 17:00:00 MST 1969 UNDEFINED default [Thu May 04 04 Exceeded cadence2 Distcp job application_1681360813939_33172 MAPREDUCE Thu May 4 04:32:18 MST 2023 Wed Dec 31 17:00:00 MST 1969 UNDEFINED default [Thu May 04 04 Exceeded cadence2 Oozie Job on Vip 0517949-230412214950046-oozie-oozi-W Shell-Action Thu May 4 04:32:18 MST 2023 Wed Dec 31 17:00:00 MST 1969 RUNNING default [Thu May 04 04 cadence2 PI-job application_1681360775209_1286 MAPREDUCE Thu May 4 11:30:15 UTC 2023 Thu May 4 11:30:27 UTC 2023 SUCCEEDED default Fine gcsidle2 Spark-job application_1681360775209_1288 SPARK Thu May 4 11:31:18 UTC 2023 Thu May 4 11:31:24 UTC 2023 SUCCEEDED default Fine gcsidle2 Spark Python Pi-job application_1681360775209_1289 SPARK Thu May 4 11:31:49 UTC 2023 Thu May 4 11:31:57 UTC 2023 SUCCEEDED default Fine gcsidle2 Distcp job application_1681360775209_1290 MAPREDUCE Thu May 4 11:32:19 UTC 2023 Thu May 4 11:32:27 UTC 2023 SUCCEEDED default Fine gcsidle2 Oozie Job on Vip 0002335-230419024434725-oozie-oozi-W Shell-Action Thu May 4 11:32:19 UTC 2023 Thu May 4 11:32:27 UTC 2023 SUCCEEDED default gcsidle2   If you check the field "FinalState" it is only picking up "SUCCEEDED" wherein other events also have UNDEFINED and RUNNING, it is not picking up those. ... View more