Dashboards & Visualizations

How to calculate concurrent running builds in Jenkins and team city using Splunk?

Hemnaath
Motivator

Hi All,

Good Morning, I have requirement to create a dashboard to display the total number of concurrent running builds in Jenkins and team city. Need 4 panels to display total number of builds with time chart and single value visualization. Already we have 9 panels to display Build count, Build duration and Queue Time by Day in place.

Details about the Panels Type and Query:

1) TC Build Duration Per day

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | time chart span =1h sum(duration)

2) TC Build count & Duration:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | time chart span =1h count (duration)

3) Jenkins Build & Queue Time By Day:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | time chart span = 1h sum(Job_duration) sum(Queue_time)

4) Jenkins Build Count & Duration & Queue Time:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | time chart span = 1h sum(Job_duration) sum(Queue_time) count

5) TC Build Count: Single Value Visualization:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | stats count

6) TC Build Count in seconds: Single Value Visualization:

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” | transaction buildId keepevicted =true | sum (duration)

7) Jenkins Build Count: Single Value Visualization:

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats count

😎 Jenkins Build Duration in seconds : Single Value Visualization

Index =JK-App sourcetype=* event_tag =”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats sum (Job_duration)

9) Jenkins Queue Time in seconds:

Index =JK-App sourcetype=* event_tag
=”Job_event” type=”completed” build_number = “*” | dedup host build_url sortby - _time | search ‘UTC_to_local_time(Job_started_at)’ | stats sum (queue_time)

Now need to create additional panels to display the concurrent running build Jobs in Jenkins and Team city. Kindly guide me how to do that.

rock_kang
New Member

@Hemnaath have you resolved it? I have a similar request, can you guide me on this. thanks

0 Karma

rock_kang
New Member

@Hemnaath have you resolved it? I have a similar request, can you guide me on this. thanks

0 Karma

koshyk
Super Champion

Assuming, you have buildId separate for each task, the Sample data I've used is to produce multiple pairs of samples with buildId different

[2019-05-20 13:41:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280844')}
[2019-05-20 13:41:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:45:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
[2019-05-20 13:42:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280850')}
[2019-05-20 13:42:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280850'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:46:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280850'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
[2019-05-20 13:43:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280851')}
[2019-05-20 13:43:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280851'). started 2019-05-20 13:41:56,574, running }
[2019-05-20 13:44:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2662'  buildId='43280852'). started 2019-05-20 13:41:56,574, running }

The search query, i've come up with this is:

index=* earliest=-7d
| rex "buildServer.ACTIVITIES -(?<build_state_text>[\w\s]+)(\;|GOM)"
| eval build_state=case(like(build_state_text, "%queue%"),"queued",like(build_state_text, "%start%"),"start",like(build_state_text, "%Finish%"),"end",true(),"unknown")
| search (build_state=start OR build_state=end)
| transaction buildId startswith=(build_state=start) endswith=(build_state=end) keepevicted=true
| where closed_txn=0
| stats count

The logic is
- Find the build_state for each events
- Filter out ONLy the started and Ended ones
- do clubbing (transaction) of these to find which ones are still running (i.e not closed)
- Do count of such unclosed transactions.

0 Karma

Hemnaath
Motivator

Sample Data. Please do guide on how to calculate the concurrent running Jobs per 1 hours for 24 hours.

[2019-05-20 13:41:55,226] INFO -tbrains.buildServer.ACTIVITIES -Build added to queue; Queued build {Build promotion {promotion id=XXXX, Configuration={id=GOM_TopazPR_GatewaysMercuryDOCWS, internal id=bt8981}, branch=, queued, modificationId=1324733, revisions=["GOM_TopazPR_VCS" {instance id=2159, parent internal id=55, parent id=GOM_Topaz_VCS, description: "git@github.ldn.xxx.com:gom/ITEQCH_TOPAZ_REL_17.git#master"}: 88af2551bf3f40503#refs/heads/master]}, triggered by "sso_gomtools' {id=xxxx}" (##type = 'SnapshotDependency' userId= 'xxxx' origin = 'rest' buildTypeId = 'bt2657' buildId = '43280844')}

[2019-05-20 13:41:56,674] INFO -tbrains.buildServer.ACTIVITIES -Build started; GOM / TopazPR / SystemEntityFeatureApi {id=GOM_TopazPR_SystemEntityFeatureApi, internal id =bt32932} #360 {promotion id=43279568, branch=<default>, agent="Linux RHEL 6.4 [xxxxldap] GOM-agent-01" {id=1376}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:41:56,574, running }

[2019-05-20 13:45:36,974] INFO -tbrains.buildServer.ACTIVITIES -Finished build  GOM / TopazPR / SystemTSProxy {id=GOM_TopazPR_SystemTSProxy, internal id=bt4858} #3756 {promotion id=43280653, branch=<default>, agent="Linux RHEL 6.4[xxxx] GOM-agent-17" {id=1876}, triggered by "'sso_gomtools' {id=4809}" (##type ='SnapshotDependency' userId='XXX' origin='rest' buildTypeId='BT2657' buildId='43280844'). started 2019-05-20 13:44:47,974, Finished. Status "NORMAL 'Running' "}
0 Karma

zshainsky
Splunk Employee
Splunk Employee

@Hemnaath, Can you share how you got the Team City data into Splunk?

Anything helps!

0 Karma

DavidHourani
Super Champion

Hi @Hemnaath,

Great question ! there's a command built specifically for that ! have a look here :

https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Concurrency

What this command does is based on a start time and duration field (which you already have from your transaction command) you can get the concurrently running events/jobs!

Let me know if that helps,

Cheers,
David

0 Karma

Hemnaath
Motivator

Hi David thanks for your support on this, I have added the sample data so not sure how to use the sample data and build the query to fetch the concurrent running jobs, could you please guide me on this.

0 Karma

DavidHourani
Super Champion

sure thing, should be something like this :

Index = TC-App sourcetype =tc:srv “Build started” OR “Finished build” 
| transaction buildId keepevicted =true 
| concurrency duration=duration 

Let me know if this helps and upvote if you find it helpful ^^
Cheers,
David

0 Karma

Hemnaath
Motivator

Hi All, Can anyone kindly guide me on this.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...