How to calculate concurrent running builds in Jenkins and team city using Splunk?

[Hemnaath]

Hi All,

Good Morning, I have requirement to create a dashboard to display the total number of concurrent running builds in Jenkins and team city. Need 4 panels to display total number of builds with time chart and single value visualization. Already we have 9 panels to display Build count, Build duration and Queue Time by Day in place.

Details about the Panels Type and Query:

1) TC Build Duration Per day

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | time chart span =1h sum(duration)

2) TC Build count & Duration:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | time chart span =1h count (duration)

3) Jenkins Build & Queue Time By Day:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | time chart span = 1h sum(Job_duration) sum(Queue_time)

4) Jenkins Build Count & Duration & Queue Time:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | time chart span = 1h sum(Job_duration) sum(Queue_time) count

5) TC Build Count: Single Value Visualization:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | stats count

6) TC Build Count in seconds: Single Value Visualization:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | sum (duration)

7) Jenkins Build Count: Single Value Visualization:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats count

😎 Jenkins Build Duration in seconds : Single Value Visualization

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats sum (Job_duration)

9) Jenkins Queue Time in seconds:

Index =JK-App sourcetype=* event_tag
=”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats sum (queue_time)

Now need to create additional panels to display the concurrent running build Jobs in Jenkins and Team city. Kindly guide me how to do that.

[rock_kang]

@Hemnaath have you resolved it? I have a similar request, can you guide me on this. thanks

[rock_kang]

@Hemnaath have you resolved it? I have a similar request, can you guide me on this. thanks

[koshyk]

Assuming, you have buildId separate for each task, the Sample data I've used is to produce multiple pairs of samples with buildId different

[2019-05-20 13:41:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280844')}
[2019-05-20 13:41:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:45:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
[2019-05-20 13:42:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280850')}
[2019-05-20 13:42:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280850'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:46:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280850'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
[2019-05-20 13:43:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280851')}
[2019-05-20 13:43:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280851'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:44:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2662'  buildId='43280852'). started 2019-05-20 13:41:56,574, running }

The search query, i've come up with this is:

index=* earliest=-7d
| rex "buildServer.ACTIVITIES -(?<build_state_text>[\w\s]+)(\;|GOM)"
| eval build_state=case(like(build_state_text, "%queue%"),"queued",like(build_state_text, "%start%"),"start",like(build_state_text, "%Finish%"),"end",true(),"unknown")
| search (build_state=start OR build_state=end)
| transaction buildId startswith=(build_state=start) endswith=(build_state=end) keepevicted=true
| where closed_txn=0
| stats count

The logic is
- Find the build_state for each events
- Filter out ONLy the started and Ended ones
- do clubbing (transaction) of these to find which ones are still running (i.e not closed)
- Do count of such unclosed transactions.

[Hemnaath]

Sample Data. Please do guide on how to calculate the concurrent running Jobs per 1 hours for 24 hours.

[2019-05-20 13:41:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280844')}

[2019-05-20 13:41:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:41:56,574, running }

[2019-05-20 13:45:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}

[zshainsky]

@Hemnaath, Can you share how you got the Team City data into Splunk?

Anything helps!

[DavidHourani]

Hi @Hemnaath,

Great question ! there's a command built specifically for that ! have a look here :

https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Concurrency

What this command does is based on a start time and duration field (which you already have from your transaction command) you can get the concurrently running events/jobs!

Let me know if that helps,

Cheers,
David

[Hemnaath]

Hi David thanks for your support on this, I have added the sample data so not sure how to use the sample data and build the query to fetch the concurrent running jobs, could you please guide me on this.

[DavidHourani]

sure thing, should be something like this :

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” 
| transaction buildId keepevicted =true 
| concurrency duration=duration 

Let me know if this helps and upvote if you find it helpful ^^
Cheers,
David

[Hemnaath]

Hi All, Can anyone kindly guide me on this.