Splunk DB Connect V3 - How create Automated / Programmatic creation of connections and inputs

[gjanders]

Please note that I'm answering this question in addition to posting it, if you find the answer useful please feel free to vote/award points!

---

The problem

After upgrading from Splunk DB Connect V2 to Splunk DB Connect V3 I was disappointed to find that some of the previously useful REST API's I was using to programatically create new connections/data inputs were no longer functional.

After logging a support ticket I was advised these were never supported, and therefore my request is now an "enhancement".

After quite a bit of work I did find a way to automate the DB Connect V3, it is likely unsupported by the official application but it works and provides a nice workaround until they officially support automation via REST API or similar.

Solution

Please see the answer below, comments/better solutions are welcome!

[girtsgr]

Can you give an example how to update a DB Connect input via the API? I'm looking for an option to disable a connection via a script.

[glimp]

Hi!

How did you manage to change a parameter in an existing connection?

if using the following command, i get "HTTP 409 -- An object with name=DB1_ABC already exists" :

curl -H "Content-Type: application/json" --insecure --netrc-file /export/home/scripts/splunk/.netrc --data @db1_abc.json https://hostname:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/connections

if adding the connecion name at the end of the URL, i get "HTTP 405 Method Not Allowed":

curl -H "Content-Type: application/json" --insecure --netrc-file /export/home/scripts/splunk/.netrc --data @db1_abc.json https://hostname:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/connections/DB1_AB...

[JorgeFT]

Hi @mchristopherson!

I've found this line very useful:

curl -k -u admin:changeme -H "Content-Type: application/json" -X PUT -d '{"name":"test","username":"yourusername", "password":"newPassword"}' https://dbx.splunk.dev:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/identities/t...

Which actually works and changes the password of an identity without any other steps required.

I hope it helps you. Regards!

[gjanders]

Thanks for updating this old post (2017) with clear instructions

[jbrocks]

I found the following stanza in the restmap.conf of the Splunk DB App.

[script:input_templates]
match=/db_connect/dbxproxy/templates/input
scripttype=python
handler=dbx_rh_proxy.ProxyManager
requireAuthentication = true
capability.get=db_connect_read_dbinput_template
capability.put=db_connect_update_dbinput_template
capability.delete=db_connect_delete_dbinput_template

So I think it should be possible to use a template input via REST. But how do I figure out what the JSON object should look like and what parameters are expected?

[gjanders]

Just noting this down as my old way of creating inputs.conf for DB Connect which is now out-dated...

Fortunately the URL / REST API for creating new inputs is not disabled via DB connect V3:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs

Continues to work via POST requests, however creating a database input via REST API will not show up in the GUI and if you do get it to show in the GUI it does not mean the DB Connect task server will run your input, therefore there are multiple additional steps:

• Create a checkpoint file
• Reload the DB connect task server
• Reload the DB connect inputs

The first I achieved through some simple python code:

#As of DB connect v3 we require checkpoint files to exist, furthermore full stops becomes underscores in the file naming
theName = dbinputName.lower().replace(".","_")
checkpointFile = "/opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect/" + theName
if os.path.isfile(checkpointFile):
    print "Checkpoint file exists, location %s , doing nothing" % (checkpointFile)
else:
    aFile = open(checkpointFile, 'w')
    aFile.writelines('{"value":"1970-01-01 00:00:00.00","appVersion":"3.0.0","columnType":93,"timestamp":"1970-01-01T00:00:00.000+11:00"}')
    aFile.close()

Note that creating a zero sized file does not work, the above input is designed for my inputs, you will need to test to ensure you have the correct line above.
Also note that the checkpoint file must be lowercase and any full stops ( . ) are replaced with underscore ( _ ) on the filesystem.

Triggering the admin server reload:
https://localhost:8089/servicesNS/nobody/search/admin/server/_reload

Triggering the database input reload:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload

Once all these are done I have both a connection & a database input all without clicking via the GUI, which is important as we do intend to have hundreds of database inputs...

Note that the admin/server/_reload URL triggers the DB Connect task server to restart itself...

[KOJL]

Hi!
First off, thank you for contributing your insights into the workings of the DB connect app. 

I have developed a collection of python3 scripts to automate adding new databases and inputs to db connect through the rest api. 

But I have run into the issues you describe in this post when starting the testing phase. 
The connections and inputs are generated in the app and are visable through the UI. 
The timestamp file is also present in the folder :

/opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect/

However, the data for the inputs are not in the index. 
After trying the solution that you describe for this i have run into some issues that i hope someone can answer. 

When trying to contact the url for reloading the inputs: 
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload
I only get a 404 status in return, leading me to think that this enpoint is no longer available through the rest api?

I'm running version 3.5.1 of db connect.

I can reach the : 
https://localhost:8089/servicesNS/nobody/search/admin/server/_reload
And the task server url with curl, but not through python requests for some reason (503 error)
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/taskserver


But in the end the inputs generated over the api does not seem to run and i'm guessing it is becouse I dont get the reload on inputs.
When running the query for the input through the UI i do get data back so i belive all parameters are present and correct for the input,identity and connection.

I guess I could restart the splunk instance all togheter but that would be a last resort kind of solution that i rather avoid. 

Update: 
The data was there,just in another index then expected due to splunk baseline configs.

As for the endpoints for reloading the task server, all it needed was a bit of time. 
If you are planning to reload the taskserver and hit the admin reload endpoint in the same script, make sure to add in breaks as it takes a couple of seconds for the service to reload and if you hit the reload task server endpoint right after the admin reload url, you will get the 503 error. 

[gjanders]

While I'm not completely sure what the new endpoints might be, one thing you might try doing is doing an update on the UI and watching the splunkd_access.log file in $SPLUNK_HOME/var/log/splunk to see if you can tell which endpoints are used

You might be able to determine the endpoint required to update the config from there...

[KOJL]

That is a nifty trick, thank you.
I did update my post with the solution to my issue, but I can't say if it's necessary to both hit the admin reload or just restarting the task server is enough. 

Doing both will at least make your inputs run. 

Once again thank you for taking the time to help out. 

[saranya_fmr]

Hi @garethatiag ,

I also wanted to automate the creation of DB Connection in DBX3 and I tried the below but it didnt seem to work:

1) Updated via CLI identities.conf and db_connections.conf with the new DB details
2) Did a reload on the host
curl -k -u admin:password https://myhost:8089/servicesNS/nobody/splunk_app_db_connect/configs/conf-db_connections/_reload

Is there anything I'm missing out? Could you please guide..

[wlaenig]

i believe identity.dat is in the certs directory is the encryption key with aes-256. however i have not able to decrypt the encrypted password.

[sloshburch]

@saranya_fmr - Something just crossed my mind that is tangential but could be extremely important for you: If you make the call with the REST API, won't the URI request appear in the _internal logs? Therefore, the username and password from the call would appear in clear text on the filesystem in the respective Splunk log AND therefore in the _internal index?

If we're lucky, those parameters are hashed or obfuscated as part of the call and don't appear but since you're not using splunkweb to make the call (and therefore no javascript to do that for you) it's possible they are not secure.

I would suggest validating that because you'll wanna make sure there's a usage approach that is secure before getting too far into implementing this way.

[gsrivastava]

@garethatiag, As far as I know we cannot create a new identity by editing the identities.conf file as the password is stored in encrypted format and the encryption key is not known.

If identity cannot be created, how can we create a new connection by editing the db_connections.conf file?

So by your solution, Can we create a new connection with a new identity?

[gjanders]

When I first automated this I could use:
/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/

So https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/ or similar with a POST request and that would have created the username/password in the appropriate encrypted format. Unfortunately newer versions (3.x) of this application removed that REST endpoint and I'm unsure if they have allowed an equivalent one so far.

When I last checked I could not find the REST API to create identities, I'm hoping the DB Connect team intend to re-add or at least allow a programmatic way of doing this...perhaps send them an enhancement request via the support portal?

[gjanders]

This looks a bit different to what I have done:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload

And:
https://localhost:8089/servicesNS/nobody/search/admin/server/_reload

Can you try those URL's? In particular the second one reloads the task server which should force it to reload the connections...

[saranya_fmr]

Hi @garethatiag ,

Did you create DB coonection and Identity via CLI initially and then did a reload via Splunk RESTAPI? What was your approach?

Im only creating a DB connection for DB Data Query and I do not want to create DB Outputs or inputs so reload inputs URL is not needed for me I think:

However I did try both the URLs that you suggested , reload task-server and reload inputs but it didnt work 😞
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload

I see the connection in UI but it gives me an error as seen in screenshot:

The splunk_app_db_connect_server.log shows below error:

2017-08-03 08:33:46.048 -0400 [dw-49 - GET /api/connections/SPOTFIRE60_DEV/status] ERROR io.dropwizard.jersey.errors.LoggingExceptionMapper - Error handling a request: 94204c89a78800be
java.lang.IllegalArgumentException: Illegal base64 character 5f
at java.util.Base64$Decoder.decode0(Base64.java:714)
at java.util.Base64$Decoder.decode(Base64.java:526)
at java.util.Base64$Decoder.decode(Base64.java:549)
at com.splunk.dbx.crypto.AESUtils.decrypt(AESUtils.java:75)

[gjanders]

If the connection does not work in the UI then the REST API will not help here, I would suggest trying to get it working manually first...

[saranya_fmr]

Hi @garethatiag ,

I noticed that the password is not getting encrypted even after Splunk restart was the root cause.
How does the password get encrypted if restart is not the solution?

[gjanders]

I'm using the URL:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/

And posting in the values for name/username/password and that takes care of any password encryption for me...

[saranya_fmr]

Hi @garethatiag ,

This URL doesnt update password/username for DB Connect V3. I assume it works only for DBX V2:

https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/

[gjanders]

@saranya_fmr Unfortunately you are correct, I think I used that url for DBX 2 and now when I re-test with DBX 3 I cannot get that to change, I tested:
/servicesNS/nobody/splunk_app_db_connect/properties/identities
And also:
/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/

And neither work...perhaps an enhancement request to the DB connect team?
It is frustrating that it cannot be automated...