Splunk fetching node as jenkins master, instead of actual slave for pipeline job builds. why?

[rakesh635]

I am using jenkins's splunk plugin version 1.6.3(latest). I have configured no executor in master, so no possibility for jobs to build in master. Have only one node configured and jobs needs to be running in that node only. From console log, i can see its running in slave node, but in splunk index, i could see its node value is master.
This case is only for pipeline jobs, but not for freestyle jobs.

[txiao_splunk]

the pipeline itself (groovy code of Jenkinsfile) is always ran on master, the node step and steps inside node closure can run on agent nodes. plugin 1.7.1 version added exec_node info to track the node step execution info

index=jenkins_statistics "stages{}.children{}.exec_node"=X

[neluvasilica]

agree with the fact that pipeline is run on master but, as we mentioned, we are interested in information about node steps execution information as well. reason for this is we want to find out if our Jenkins agents are being over/under-utilized and I didn't seem to find a way of displaying that information in the Splunk app for the pipeline builds.
silly question, the X in your query should be replaced by the master's hostname? are there quotes required?
just ran the query for one of the hosts and no results unfortunately.

[txiao_splunk]

"stages{}.children{}.exec_node"=X where X is agent node name, where the node step got executed. It is added in jenkins plugin 1.7.1 released at Dec 7, 2018.
If you have latest app, the "Build History" tab of agent will include the "stages{}.children{}.exec_node" info. the underlaying spl used is

index=jenkins_statistics host="FQDN-OF-JENKIS-MASTER" event_tag=job_event ( node="JENKINS-AGENT-NAME" OR "stages{}.children{}.exec_node"="JENKINS-AGENT-NAME" )| dedup build_url sortby -_time

[neluvasilica]

thank you, the spl works. is there a way of displaying the "number of builds" across multiple nodes/slaves over a specific period of time when using pipelines? something similar to visualizing "Build Distribution across Slaves" when for non-pipeline builds?
this would be very helpful in identifying if the number of nodes is over or under provisioned.
in other words, the "build history" in v1.7.1 is fine for individual nodes but is there a way of visualizing the builds history for all nodes connected to a pipelined master?

[neluvasilica]

@txiao_splunk are you aware of a way of achieving what I have described in my last comment? can I achieve this via a custom dashboard?

[txiao_splunk]

of course you can use custom dashboard, you could try run the sample spl

 index=jenkins_statistics host="FQDN-OF-JENKIS-MASTER" event_tag=job_event type=started ( node="JENKINS-AGENT-NAME" OR "stages{}.children{}.exec_node"="JENKINS-AGENT-NAME" )
| rename "stages{}.children{}.exec_node" as exec_node
| eval build_on_node=coalesce(exec_node, node)
| timechart count by build_on_node

Then "Save as Report". You can also create a custom dashboard with node filter and time filter, please checkout the free book https://www.splunk.com/goto/book and dashboard doc https://docs.splunk.com/Documentation/Splunk/7.2.3/Viz/WebFramework

[neluvasilica]

thank you. that is very useful. So in my syntax:

    index=jenkins_statistics host="FQDN-OF-JENKINS_MASTER" event_tag=job_event  
| rename "stages{}.children{}.exec_node" as exec_node 
| eval build_on_node=coalesce(exec_node, node)  
| chart count by build_on_node useother=f 
| sort -count

I can see my jenkins slaves and the "count". However I am not too sure what "count"represents in this case. The number is quite high. For instance there are 60 events and 19420 for "count".Is there a way to display the number of builds in the count field instead of that high number which I am not sure what it represents?
appreciate your help.

[txiao_splunk]

I think it counted pipeline steps, maybe we can try mvdedup, e.g.

index=jenkins_statistics host="FQDN-OF-JENKINS_MASTER" event_tag=job_event type=completed
 | rename "stages{}.children{}.exec_node" as pipeline_nodes
 | eval exec_nodes=mvdedup(pipeline_nodes)
 | eval build_on_node=coalesce(exec_nodes, node)  
 | chart count by build_on_node useother=f 
 | sort -count

[neluvasilica]

@txiao_splunk I have a strange case where for a Jenkins master stages are still attributed to the master instead of the slaves they are running on. Version of Jenkins and Splunk plugins(including the Extension plugin) are exactly the same on 10 masters, 9 of them correctly attribute the stages to the nodes they're running on but one doesn't. The builds structure is more or less the same so I don't think that can be the issue. The build's console definitely has entries like: Running on "hostname bla" but when I check splunk, stages{}.children{}.exec_node is always attributed to the master. Any idea of what can be wrong?

[txiao_splunk]

Could you kindly check pipline related plugins (such as workflow-job, workflow-support) versions to make sure they are consistent with other masters

[neluvasilica]

workflow related plugins(workflow-job, support, cps) seem to be exactly the same on all masters.
workflow-job:
Plugin-Version: 2.35
Hudson-Version: 2.138.4
Jenkins-Version: 2.138.4
Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow
-support:3.3
workflow-cps:
Plugin-Version: 2.74
Hudson-Version: 2.138.4
Jenkins-Version: 2.138.4
Plugin-Dependencies: workflow-api:2.36,workflow-scm-step:2.4,workflow-
step-api:2.20,workflow-support:3.3,scm-api:2.2.6,script-security:1.62
,structs:1.20,support-core:2.43;resolution:=optional,ace-editor:1.0.1
,jquery-detached:1.2.1

[philsegal]

Did you get anywhere with this? We are experiencing the same issue and trying to integrate splunk with our pipeline builds?

[neluvasilica]

we are having exactly the same issue where pipeline builds are accredited to the master despite running on multiple slaves. basically when I go to the "Jenkins slaves" section and look at Build Distribution across Slaves, all builds appear as running on the master which is not correct.
@txiao_splunk do you have any suggestions?

[txiao_splunk]

app for jenkins 1.0.8 changed wording slaves to nodes and updated the agent build history tab to include node step execution (require plugin 1.7.1), but unfortunately no change for "Build Distribution" stats