topic Re: How to calculate concurrent running builds in Jenkins and team city using Splunk? in Dashboards & Visualizations

How to calculate concurrent running builds in Jenkins and team city using Splunk?

Hemnaath — Wed, 30 Sep 2020 00:36:07 GMT

Hi All,

Good Morning, I have requirement to create a dashboard to display the total number of concurrent running builds in Jenkins and team city. Need 4 panels to display total number of builds with time chart and single value visualization. Already we have 9 panels to display Build count, Build duration and Queue Time by Day in place.

Details about the Panels Type and Query:

1) TC Build Duration Per day

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | time chart span =1h sum(duration)

2) TC Build count & Duration:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | time chart span =1h count (duration)

3) Jenkins Build & Queue Time By Day:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | time chart span = 1h sum(Job_duration) sum(Queue_time)

4) Jenkins Build Count & Duration & Queue Time:

5) TC Build Count: Single Value Visualization:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | stats count

6) TC Build Count in seconds: Single Value Visualization:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | sum (duration)

7) Jenkins Build Count: Single Value Visualization:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats count

😎 Jenkins Build Duration in seconds : Single Value Visualization

9) Jenkins Queue Time in seconds:

Index =JK-App sourcetype=* event_tag
=”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats sum (queue_time)

Now need to create additional panels to display the concurrent running build Jobs in Jenkins and Team city. Kindly guide me how to do that.

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

Hemnaath — Mon, 20 May 2019 09:22:32 GMT

Hi All, Can anyone kindly guide me on this.

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

DavidHourani — Mon, 20 May 2019 11:16:02 GMT

Hi @Hemnaath,

Great question ! there's a command built specifically for that ! have a look here :

https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Concurrency

What this command does is based on a start time and duration field (which you already have from your transaction command) you can get the concurrently running events/jobs!

Let me know if that helps,

Cheers,
David

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

Hemnaath — Wed, 30 Sep 2020 00:37:05 GMT

Sample Data. Please do guide on how to calculate the concurrent running Jobs per 1 hours for 24 hours.

[2019-05-20 13:41:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280844')}

[2019-05-20 13:41:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:41:56,574, running }

[2019-05-20 13:45:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

Hemnaath — Tue, 21 May 2019 06:00:54 GMT

Hi David thanks for your support on this, I have added the sample data so not sure how to use the sample data and build the query to fetch the concurrent running jobs, could you please guide me on this.

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

DavidHourani — Tue, 21 May 2019 06:49:46 GMT

sure thing, should be something like this :

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” 
| transaction buildId keepevicted =true 
| concurrency duration=duration

Let me know if this helps and upvote if you find it helpful ^^
Cheers,
David

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

koshyk — Tue, 21 May 2019 09:39:03 GMT

Assuming, you have buildId separate for each task, the Sample data I've used is to produce multiple pairs of samples with buildId different

[2019-05-20 13:41:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280844')}
[2019-05-20 13:41:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:45:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
[2019-05-20 13:42:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280850')}
[2019-05-20 13:42:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280850'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:46:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280850'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
[2019-05-20 13:43:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280851')}
[2019-05-20 13:43:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280851'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:44:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2662'  buildId='43280852'). started 2019-05-20 13:41:56,574, running }

The search query, i've come up with this is:

index=* earliest=-7d
| rex "buildServer.ACTIVITIES -(?<build_state_text>[\w\s]+)(\;|GOM)"
| eval build_state=case(like(build_state_text, "%queue%"),"queued",like(build_state_text, "%start%"),"start",like(build_state_text, "%Finish%"),"end",true(),"unknown")
| search (build_state=start OR build_state=end)
| transaction buildId startswith=(build_state=start) endswith=(build_state=end) keepevicted=true
| where closed_txn=0
| stats count

The logic is
- Find the build_state for each events
- Filter out ONLy the started and Ended ones
- do clubbing (transaction) of these to find which ones are still running (i.e not closed)
- Do count of such unclosed transactions.

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

zshainsky — Fri, 23 Aug 2019 19:29:51 GMT

@Hemnaath, Can you share how you got the Team City data into Splunk?

Anything helps!

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

rock_kang — Fri, 09 Oct 2020 08:27:14 GMT

@Hemnaath have you resolved it? I have a similar request, can you guide me on this. thanks

Re: How to calculate concurrent running builds in Jenkins and team city using Splunk?

rock_kang — Fri, 09 Oct 2020 08:32:52 GMT

@Hemnaath have you resolved it? I have a similar request, can you guide me on this. thanks