Hi,
I am trying to implement a dashboard in splunk that presents data basing on Jenkins events. I use Splunk App for Jenkins and Splunk Jenkins plugin to send the events data.
Idea of the dashboard is to display data about running active checks for Pull Requests in associated GitHub repository. Checks are designed in Jenkins in a way to have a trigger job which calls downstream jobs. In a dashboard, I'd like to present basic info about pull request and results of test coming from downstream jobs. Unfortunately, event for trigger job does not provide info about its downstream jobs.
I collect it in such a way:
index="jenkins_statistics" event_tag=job_event build_url="job/test_trigger/10316*" | eventstats latest(type) as latest_type by build_url, host | where latest_type="started" | eval full_build_url=https://+host+"/"+build_url | eval started=tostring(round((now() - strptime(job_started_at, "%Y-%m-%dT%H:%M:%S.%N"))/60,0)) + " mins ago" | append [ search index="jenkins_statistics" event_tag=job_event upstream="job/test_trigger/10316*" trigger_by="*test_trigger*"]
where subsearch is appended to provide information about downstream jobs.
I checked https://plugins.jenkins.io/splunk-devops/#plugin-content-i-am-using-upstreamdownstream-jobs-how-can-... but this does not fit to my case, as tests are represented by downstream jobs and I'd like to have an actual data of them to display only failed ones in the dashboard.
I had a plan to create custom python command (https://dev.splunk.com/enterprise/docs/devtools/customsearchcommands/createcustomsearchcmd/) which will:
- parse data from downstream jobs events
- create new fields in trigger job event basing on above
- finally, return only trigger job event
Having that, I could be able to have all the interesting data in one event per Pull Request and format the table at the end.
Unfortunately, it does not work as I wanted. It makes Splunk hanging even for single Pull Request case (1 trigger event + 20 downstream events). This python script iterates over the events twice (firstly to process downstream jobs and secondly to find trigger and add new fields there and return that). I am afraid that it is not a best approach. Example script based on https://github.com/splunk/splunk-app-examples/blob/master/custom_search_commands/python/customsearch... is presented below:
#!/usr/bin/env python
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import sys
sys.path.append(os.path.join('opt', 'splunk', 'etc', 'apps', 'splunk_app_jenkins', 'bin', 'libs')) # splunklib
from splunklib.searchcommands import dispatch, EventingCommand, Configuration, Option
@Configuration()
class ProcessTestsCommand(EventingCommand):
""" Filters and updates records on the events stream.
##Example
:code:`index="*" | processtests
"""
@staticmethod
def get_win(tests_list, pr_num):
for test in tests_list:
if test.get('job_name') == 'build-win' and test.get('upstream') == f"job/test_trigger/{pr_num}":
return test.get('job_result') if test.get('job_result') else ''
@staticmethod
def get_ubuntu(tests_list, pr_num):
for test in tests_list:
if test.get('job_name') == 'build-ubuntu' and test.get('upstream') == f"job/test_trigger/{pr_num}":
return test.get('job_result') if test.get('job_result') else ''
@staticmethod
def get_failed_tests(tests_list, pr_num):
failed_tests = []
failed_string = ''
for test in tests_list:
if test.get('upstream') == f"job/test_trigger/{pr_num}" and test.get('job_result') != 'SUCCESS':
failed_tests.append(test)
for failed_test in failed_tests:
name = failed_test.get('job_name').split('/')[-1]
status = failed_test.get('job_result')
failed_string += f"{name} {status}\n"
return failed_string
def transform(self, records):
tests = []
for record in records:
if record.get('job_name') != 'test_trigger':
tests.append(record)
for record in records:
if record.get('job_name') == 'test_trigger':
pr_num = record.get('build_number')
build_win = self.get_win(tests, pr_num)
build_ubuntu = self.get_ubuntu(tests, pr_num)
failed_tests = self.get_failed_tests(tests, pr_num)
record['win'] = build_win
record['ubuntu'] = build_ubuntu
record['failed_tests'] = failed_tests
yield record
return
dispatch(ProcessTestsCommand, sys.argv, sys.stdin, sys.stdout, __name__)
Goal is to have all the data (from trigger and downstream jobs) in a single event representing trigger job. Do you have any ideas, what could be the better way to achieve that? I also thought about dropping this subsearch and collecting the data of downstream jobs via GitHub or Jenkins API inside Python script, but this is not preferred (API may return some malformed data, I could be limited by api hitting limits).
Appreciate any help.
