@rich7177 one question here is there a way to restrict this search with upper case and lower case scenarios. index=aap_prod sourcetype="HDP:PROD:OOZIE" (":start:] with user-retry state" OR "@end***]Action updated in DB!") when I use this @end*** it also returns results with END_IMPORT*** , which is an event detail for failed jobs. https://answers.splunk.com/answers/372033/filter-data-with-upper-case-and-lower-case-scenari.html?minQuestionBodyLength=80 ... View more

@rich7177 Thank You so much for your help, ... View more

@rich7177 I think I got the problem and here is what was happening. I was using using the renamed, names for "JOB ID" and was causing an issue with JOB ID not matching in the two grouped events. Using the following query below , this works but some how the start and the end time is still an issue. I cant get them right in this query, is it because the two events are grouped and its not able to get those right? index=aap_prod sourcetype="HDP:PROD:OOZIE" (":start:] with user-retry state" OR "@end***]Action updated in DB!") | rex "TOKEN\[\] APP\[(?<JobName>[^\]]*)"| rex "ACTION\[[^\@]*(?<Action>[^\d\]]*)" | rex "JOB\[?(?<JobId>[\d-]+)-" | transaction JobId startswith=":start:] with user-retry state" endswith="@end***]Action updated in DB!" |table JobName,JobId, duration Thanks. ... View more

@rich7177 Thanks for the reply. To answer your questions: 1) So, if you run just index=aap_prod sourcetype="HDP:PROD:OOZIE" do you get fields like JobId and all that? (Might have to change to "Verbose mode" instead of "Fast" or "Smart" mode, in the upper right). : Yes it does give a JobID i verbose mode and seems was clubbing events based on that. but the weird part is its clubbing two events with different ID. 2/19/16 2016-02-19 12:05:02,482 INFO ActionStartXCommand:543 - SERVER[p01bdl841.com] USER[hdfs] GROUP[-] TOKEN[] APP[WF_Pig_SIS_RAW_INGEST_ES] JOB[0001685-160210171129236-oozie-oozi-W] ACTION[0001685-160210171129236-oozie-oozi-W@:start:] Start action [0001685-160210171129236-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2016-02-19 12:09:06,932 INFO ActionStartXCommand:543 - SERVER[p01bdl841.com] USER[hdfs] GROUP[-] TOKEN[] APP[WF_CASggflume] JOB[0001658-160210171129236-oozie-oozi-W] ACTION[0001658-160210171129236-oozie-oozi-W@end] [0001658-160210171129236-oozie-oozi-W@end]Action updated in DB! 2) If you run the index=aap_prod sourcetype="HDP:PROD:OOZIE" | transaction ... one, it group them correctly? If so, check for duration.,: Seems have an issue grouping them correctly. But when i use the | transaction "JOB ID" which is the "JOB ID" extracted from the custom rex, what it does is apart from startwith and endswith events, it also groups the processes associated for that JOB ID(all the steps/processes which ran between the start event and the end event) The count is correct though. Thanks. ... View more

@rich7177 It was grouping two events in one even with the rex written after the transaction. is that what you were referring to.. if thats the one , its grouping start and end events as one event. ... View more

@rich7177 Thank you so much for this awesome reply. here is what is happening. I tried to use your part index=aap_prod sourcetype="HDP:PROD:OOZIE" | transaction JobID startswith=":start:] with user-retry state" endswith="@end***]Action updated in DB!" | search duration!=0 | rex "TOKEN\[\] APP\[(?<JobName>[^\]]*)"| rex "ACTION\[[^\@]*(?<Action>[^\d\]]*)" | rex "JOB\[?(?<JobId>[\d-]+)-"|rename JobId as "Job ID",JobName as "Job Name" | table "Job Name", "Job ID",StartTime , EndTime ,duration ************above part works except the start and end time******************* if I add below mentioned part it does not return any data in the table | stats earliest(_time) as StartTime, latest(_time) as EndTime | eval StartTime=strftime(StartTime, "%c")|eval EndTime=strftime(EndTime, "%c") | eval JobRunTime=tostring(duration, "duration") The previous rex works fine in this one. just that the start and end times are not displayed in the table. Events : ********************************* 2/18/16 7:11:46.526 PM 2016-02-18 19:11:46,526 INFO ActionStartXCommand:543 - SERVER[p01bdl841.aap.csaa.com] USER[hdfs] GROUP[-] TOKEN[] APP[WF_danlaw_journey_hive] JOB[0001550-160210171129236-oozie-oozi-W] ACTION[0001550-160210171129236-oozie-oozi-W@end] [***0001550-160210171129236-oozie-oozi-W@end***]Action updated in DB! 2/18/16 7:05:01.882 PM 2016-02-18 19:05:01,882 INFO ActionStartXCommand:543 - SERVER[p01bdl841.aap.csaa.com] USER[hdfs] GROUP[-] TOKEN[] APP[WF_danlaw_journey_hive] JOB[0001550-160210171129236-oozie-oozi-W] ACTION[0001550-160210171129236-oozie-oozi-W@:start:] Start action [0001550-160210171129236-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] ********************************* Thanks a ton for looking into this. ... View more

@somesoni2 We have TID and Proxyname common between both the queries ... View more

instead of append can I join it some how? ... View more

@somesoni2 that works ... View more

@vasanthmss I did check those settings earlier. It is set to default system tried playing around the settings but does work. Thanks, ... View more

@sundareshr @somesoni2 I have attached the snapshot. For each jobid its correct runtime is a row below. Job which shows runtime as Jan 14th its run time is Jan 13th ... View more

@sundareshr the timezone setting in props.conf? anything specific which you referring. Both Splunk and Hadoop infra is in the same timezone. ... View more

@somesoni2 yes for all the rows in the table ... View more

@sundareshr I think the problem is with the stats table. So whats happening is when we get the job details like Job Name, Job ID, Start Time, end Time and Job Run Time The Start Time is a day ahead in the stats table. If the stats table shows the job details for 8th its actually for 7th of Jan. Stats Table index=aap_prod sourcetype="HDP:PROD:OOZIE" (":start:] with user-retry state" OR "@end**]Action updated in DB!") | rex "TOKEN[] APP[(?[^]])"| search JobName=BOB_PAS_WF_IM_DataStore| rex "ACTION[[^\@](?[^\d]])" | rex "JOB[?(?[\d-]+)-"|rename JobId as "Job ID",JobName as "Job Name" | streamstats current=f window=2 range(_time) as JobRunTime latest(_time) as EndTime earliest(_time) as StartTime | table "Job Name", "Job ID", StartTime , EndTime ,JobRunTime|dedup "Job ID"| eval StartTime=strftime(StartTime, "%c")|eval EndTime=strftime(EndTime, "%c")|eval JobRunTime=tostring(JobRunTime, "duration")| search JobRunTime!=0 Time Chart index=aap_prod sourcetype="HDP:PROD:OOZIE" (":start:] with user-retry state" OR "@end**]Action updated in DB!") | rex "TOKEN[] APP[(?[^]])" | search JobName=BOB_PAS_WF_IM_DataStore| rex "ACTION[[^\@](?[^\d]])" | rex "JOB[?(?[\d-]+)-"|rename JobId as "Job ID" | streamstats current=f window=2 range(_time) as JobRunTime|dedup "Job ID"| search JobRunTime!=0 |timechart values(JobRunTime) by JobName ... View more

@sundareshr The range is calculated on two different events with the above mentioned time. Instead of average can we not plot the actual values in a timechart. 1/10/16 7:45:39.005 AM 2016-01-10 07:45:39,005 INFO ActionStartXCommand:543 - SERVER[p01bdl841.aap.csaa.com] USER[hdfs] GROUP[-] TOKEN[] APP[BOB_PAS_WF_IM_DataStore] JOB[0018890-150810162217214-oozie-oozi-W] ACTION[0018890-150810162217214-oozie-oozi-W@end] [0018890-150810162217214-oozie-oozi-W@end]Action updated in DB! 1/10/16 7:00:10.825 AM 2016-01-10 07:00:10,825 INFO ActionStartXCommand:543 - SERVER[p01bdl841.aap.csaa.com] USER[hdfs] GROUP[-] TOKEN[] APP[BOB_PAS_WF_IM_DataStore] JOB[0018890-150810162217214-oozie-oozi-W] ACTION[0018890-150810162217214-oozie-oozi-W@:start:] Start action [0018890-150810162217214-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] Thanks you so much for looking into this. ... View more