In general terms, yes. It's fairly easy to programmatically manipulate Splunk's configuration files. Options include (1) editing the files directly using a bit of code, (2) using the CLI ( splunk add monitor, for example), and (3) using the REST API. All of these approaches have pros/cons. I suspect that option #2 is out for a custom TA. Between options 1 and 3, it's a bit of a preference call. Both options will give you lots of flexibility.
I've not used the Jenkin's TA, but if you can figure out which entry in your configuration file needs to be updated, it should be fairly straightforward to update it.
If you want to go down the REST API path, I'd suggest starting with the REST API Tutorials - Managing Objects, if you haven't done anything like this before with Splunk. Then make your way over to the Splunk REST API Reference docs and look at the "Configuration" section.
