Monitoring Splunk

How to fetch the status of an application having multiple jobs ?

Real_captain
Path Finder

HI Team 

Can someone please help me to find how we can fetch the status of the application A1 having 5 jobs (Job1 , Job2 , Job3 , Job4 , Job5) running every day. 

Status of Application : This needs to be extracted using the query attached below: 
Planned : If current time is less than the expected time of JOB1 
OK-Running :  If Current time is between the expected time of JOB1 and expected time of JOB5 + Status of all the JOBs is either OK  or PLANNED
KO-FAILED : if Current time is between the expected time of JOB1 and expected time of JOB5 + Status of any the 1 JOBs is either KO. 

Query used today to fetch the status of each job in the application: 

index = ABC ( TERM(JOB1) OR TERM(JOB4) OR TERM(JOB2) OR TERM(JOB3) OR TERM(JOB5) OR TERM(JOB6) OR TERM(JOB7) ) ("- ENDED" OR "- STARTED" OR "ENDED - ABEND")
| eval Function = case(like(TEXT, "%ENDED - ABEND%"), "ABEND" , like(TEXT, "%ENDED - TIME%"), "ENDED" , like(TEXT, "%STARTED - TIME%"), "STARTED")
| eval DAT = strftime(relative_time(_time, "+0h"), "%d/%m/%Y") , {Function}_TIME=_time
| rename DAT as Date_of_reception
| stats max(Date_of_reception) as Date_of_reception max(ENDED_TIME) as ENDED_TIME max(STARTED_TIME) as STARTED_TIME max(ABEND_TIME) as ABEND_TIME by JOBNAME
| inputlookup append=t ESES_Job_MIFID_PPE.csv
| stats values(*) as * by JOBNAME
| eval DAY_OF_WEEK = strftime(strptime(Date_of_reception, "%d/%m/%Y"), "%A") ,today = strftime(1743030000, "%Y-%m-%d") , TO_DAY = strftime(strptime(today, "%Y-%m-%d"), "%A") , Diff=ENDED_TIME-STARTED_TIME
| rename STARTED_TIME as START_TIME1 , ENDED_TIME as END_TIME1 , ABEND_TIME as ABEND_TIME1
| eval diff_time = tostring(Diff , "duration"), diff_time_1=substr(diff_time,1,8) ,
START_TIME = Date_of_reception." ".strftime((START_TIME1),"%H:%M:%S") ,
END_TIME = Date_of_reception." ".strftime((END_TIME1),"%H:%M:%S") , END_TIME2 = strftime((END_TIME1),"%H:%M:%S") ,
ABEND_TIME = Date_of_reception." ".strftime((ABEND_TIME1),"%H:%M:%S") , ABEND_TIME2 = strftime((ABEND_TIME1),"%H:%M:%S") ,
EXPECTED_TIME = exp_time , DEADLINE_TIME = high_dl2
```EXPECTED_TIME_run = Date_of_reception." ".EXPECTED_TIME, EXPECTED_TIME_run = strptime(EXPECTED_TIME_run, "%d/%m/%Y %H:%M:%S") , TimeDiff=EXPECTED_TIME_run-now() ,
EXP_TIME_norun = if (TO_DAY = "Friday" , exp_time2 , exp_time1) ,EXPECTED_TIME_norun = today + " " + EXP_TIME_norun, EXPECTED_TIME_norun = strptime(EXPECTED_TIME_norun, "%Y-%m-%d %H:%M:%S") , TimeDiff_norun =EXPECTED_TIME_norun-now()
, Time_Diff=now() - strptime(START_TIME, "%d/%m/%Y %H:%M:%S") ```
| eval STATUS =
if(isnotnull(END_TIME2) AND (END_TIME2 <= ABEND_TIME2),"ABEND",
if(isnotnull(END_TIME2) AND (END_TIME2 <= DEADLINE_TIME),"OK",
if(isnotnull(END_TIME2) AND (END_TIME2 > DEADLINE_TIME),"BREACHED",
if(isnull(END_TIME2) AND isnull(START_TIME1) AND (TimeDiff_norun > 300),"PLANNED",
if(isnull(END_TIME2) AND isnull(START_TIME1) AND isnull(TimeDiff) AND (TimeDiff_norun < -600) AND (TimeDiff_norun >= -1800),"JOB NOT STARTED YET",
if(isnull(END_TIME2) AND isnull(START_TIME1) AND isnull(TimeDiff) AND (TimeDiff_norun < -1800),"JOB DID NOT EXECUTED",
if(isnull(END_TIME2) AND isnotnull(START_TIME1) AND (Time_Diff>600),"FAILED",
if(isnull(END_TIME2) AND isnotnull(START_TIME1) and (TimeDiff<=600),"RUNNING",
if( isnull(END_TIME2) AND isnull(START_TIME1) AND JOBNAME IN ("$JOB3" ) , "OK-Interest file is received" ,
if( isnull(END_TIME2) AND isnull(START_TIME1) AND JOBNAME IN ("$JOB6") , "OK-Mifid 2 file is received" ,
if( isnotnull(END_TIME2) AND isnotnull(START_TIME1) AND JOBNAME IN ("$JOB3" ) , "KO-Interest file Not received" ,
if( isnotnull(END_TIME2) AND isnotnull(START_TIME1) AND JOBNAME IN ("$JOB6") , "KO-Mifid 2 file Not received" ,
"WARNING"))))))))))))
| rename diff_time_1 as EXECUTION_TIME
| sort Order
| table Application,JOBNAME,Description, EXPECTED_TIME , DEADLINE_TIME , START_TIME , END_TIME ,EXECUTION_TIME, STATUS
| fillnull value="-"

 

Labels (1)
0 Karma

Real_captain
Path Finder

@ITWhisperer  

Can you please help me in this topic. 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Please do not tag me - I, like many here, volunteer my time and expertise and it is not for others to suggest what I work on. By specifically addressing people, you are also potentially excluding others who may have valuable contributions to make; it is like you don't value or are not interested in their efforts (since you haven't also directly addressed them). I imagine this can be counter-productive to resolving your issue!

Real_captain
Path Finder

@isoutamo 
is it possible to correct my splunk query to fetch the status of the application as below : 

Status of Application : This needs to be extracted using the query attached below: 
Planned : If current time is less than the expected time of JOB1 
OK-Running :  If Current time is between the expected time of JOB1 and expected time of JOB5 + Status of all the JOBs is either OK  or PLANNED
KO-FAILED : if Current time is between the expected time of JOB1 and expected time of JOB5 + Status of any the 1 JOBs is either KO. 

0 Karma

Real_captain
Path Finder

Sample events are attached. 

 

 
 

 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Have you any sample events which you can share with us? If yes, then anonymous those if/as needed.
0 Karma

Real_captain
Path Finder

@isoutamo  or Someone from Splunk Team 

Can you please help to provide me a solution for this type of result. 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Your Sample logs.txt didn't work with your query. I suppose that it's not a real sample data what you have gotten from index=.... query!
Also there is one lookup which content is unknown.
0 Karma

Real_captain
Path Finder

Splunk query + Csv file + real output data is attached. 

Real_captain_0-1743666043902.png

 

 

0 Karma

Real_captain
Path Finder

Csv file is attached 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
We don’t need an output files. Raw input events are what’s still missing.
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Sample events have been provided below, but, unfortunately, they don't match the supplied lookup and are not in a good format (fields and headers have different delimiters and are consequently not aligned well!)

0 Karma

isoutamo
SplunkTrust
SplunkTrust
I already try those after fixing delimiters etc, but SPL expects some _time which aren’t present in this example file.
0 Karma
Get Updates on the Splunk Community!

Good Sourcetype Naming

When it comes to getting data in, one of the earliest decisions made is what to use as a sourcetype. Often, ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...

Splunk App for Anomaly Detection End of Life Announcement

Q: What is happening to the Splunk App for Anomaly Detection?A: Splunk is officially announcing the ...