topic Re: wanna show the of the job as it is untill its changes its status in Splunk Search

wanna show the of the job as it is untill its changes its status

pench2k19 — Wed, 30 Sep 2020 04:12:29 GMT

Hi Team

I have following details

One of autosys job is running for 20 hours with the status recording in the logs as RUNNING recording only one event with status in the logs .i.e when it changed from the status STARTING to RUNNING.

I would like to show the same status of the Job for that 20 hours duration in the timechart .

I am using the following query, but no able to get the solution. can you please help.

You help is much appreciated.

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Sun, 16 Feb 2020 09:33:11 GMT

| search Status=STARTING AND cluster= AND host="" AND Job=*

This query can't find status change. This displays only Status=STARTING.

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Wed, 30 Sep 2020 04:13:24 GMT

@to4kawa thanks for the response, the following is my updated query

I dont mind if you edit the query the way you want but i need to see the number of jobs running with what status by the clustert at any given point of time. And the if a job duration is 3 hours for suppose, the status of the job has to be same through out the 3 hours as RUNINNG.

P S : I have to deal with only two status from this query i.e RUNNING and STARTING

@somesoni2 @woodcock @richgalloway @jkat54

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Mon, 17 Feb 2020 09:10:15 GMT

index=infra_apps sourcetype=ca:atsys:edemon:txt
| rename hostname as host
| fields Job host Autosysjob_time Status
| lookup datalakenodeslist.csv host OUTPUT cluster
| mvexpand cluster
| dedup Job Autosysjob_time host

Can you provide the result at this point?

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Mon, 17 Feb 2020 10:02:20 GMT

@to4kawa its giving me reulsts with the following as fields when i add table * at the end of above query

Autosysjob_time Job Status cluster host _raw _time

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Wed, 30 Sep 2020 04:07:29 GMT

what's Autosysjob_time ?
timechart uses _ time.
why useless field is remain?

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Mon, 17 Feb 2020 10:46:42 GMT

Thats when the autosys update the status in th records, we can use that as _time

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Mon, 17 Feb 2020 13:45:11 GMT

UPDATE:

index=infra_apps sourcetype=ca:atsys:edemon:txt 
| rename hostname as host 
| fields Job host Autosysjob_time Status 
| lookup datalakenodeslist.csv host OUTPUT cluster 
| mvexpand cluster 
| dedup Job Autosysjob_time host 
| table Autosysjob_time Job Status cluster host
| sort Autosysjob_time
| streamstats values(Status) as p_Status window=1 current=F  by cluster
| eval flag=if(Status!=p_Status,"change",NULL)
| streamstats count(eval(flag="change")) as session by cluster
| rename Autosysjob_time as _time
| bin _time span=5m
| stats count(Job) as job_count count(eval(status="STARTING")) as STARTING  count(eval(status="RUNNING")) as RUNNING count(eval(status="SUCCESS")) as SUCCESS by _time cluster session

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Wed, 30 Sep 2020 04:14:19 GMT

here is the sample events FYR :

[02/18/2020 10:37:27.6386] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: Job1
Status = STARTING host = XXXX source = /opt/CA/r1 sourcetype = ca:atsys:edemon:txt

[02/18/2020 10:39:27.6386] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: Job1
Status = RUNNING host = XXXX source = /opt/CA/r1 sourcetype = ca:atsys:edemon:txt

[02/18/2020 12:37:27.6386] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: Job1
Status = RUNNING host = XXXX source = /opt/CA/r1 sourcetype = ca:atsys:edemon:txt

the latest query i have defined is as follows

I am not sure if this is correct query to check as this query running very slow.

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Wed, 30 Sep 2020 04:14:21 GMT

small correction in the sample events

[02/18/2020 10:47:15.1318] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: STARTING JOB: CFDW_ADHOC_C_AIMSAS_D_INV_LNITEM_BILLING_CHGS_M MACHINE: XXXX

[02/18/2020 10:48:15.1318] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: RUNNING JOB: CFDW_ADHOC_C_AIMSAS_D_INV_LNITEM_BILLING_CHGS_M MACHINE: XXXX

[02/18/2020 11:57:15.1318] CAUAJM_I_40245 EVENT: CHANGE_STATUS STATUS: SUCCESS JOB: CFDW_ADHOC_C_AIMSAS_D_INV_LNITEM_BILLING_CHGS_M MACHINE: XXXX

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Tue, 18 Feb 2020 20:46:07 GMT

I am not sure Autosysjob_time , but my query is like OK.
what's wrong?

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Wed, 30 Sep 2020 04:14:41 GMT

@to4kawa Autosysjob_time is the timestamp that is available in the sample event, i see the difference in indexing time and Autosysjob_time. Confused what to populate on the x- axis.

Your query seems to be OK, but i am not able to get how many jobs are in running or Starting state at each minute by each cluster with your query.

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Wed, 19 Feb 2020 20:59:51 GMT

how many jobs are in running or Starting state at each minute by each cluster is not
show the of the job as it is untill its changes its status. which do you want?

ans1. count state=Running and state=Starting each other.
ans2. count the change like my query.

Re: wanna show the of the job as it is untill its changes its status

woodcock — Thu, 20 Feb 2020 00:31:52 GMT

You need the concurrency command. Google for examples and you will find what you need (I have provided some answer before so that may help narrow it down):
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Concurrency

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Fri, 21 Feb 2020 08:59:10 GMT

@to4kawa I would like to combine both of them.

I have an event with status=running at 10:48 as shown in the sample event, when i do the auditing for evvery 5 minutes it should get counted in the number of running jobs untill it changes it changes its job status=success at 11:57.

Re: wanna show the of the job as it is untill its changes its status

to4kawa — Fri, 21 Feb 2020 09:56:14 GMT

I'm not sure what you want to count the status.
my answer is updated. please confirm.

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Fri, 21 Feb 2020 14:42:11 GMT

@woodcock Concurrency command looking like doesnt suit to my use case....For example i want to hold the job with running status and get it counted till it changes its next status to success when i audit it with my timechart for every 5 mins.

Re: wanna show the of the job as it is untill its changes its status

woodcock — Fri, 21 Feb 2020 17:11:47 GMT

Any time you use co currency, you need to do some extra work but it is the only command that can easily do what you need. Look for answers by @woodcock and by @sideview.

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Tue, 25 Feb 2020 12:08:44 GMT

Finally, i solved my own puzzle, Thanks for everyone for spending their time here

Re: wanna show the of the job as it is untill its changes its status

pench2k19 — Tue, 25 Feb 2020 12:09:49 GMT

Finally, i solved my own puzzle, Thanks to everyone for spending their time here