All Apps and Add-ons
Highlighted

Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

SplunkTrust
SplunkTrust

Please note that I'm answering this question in addition to posting it, if you find the answer useful please feel free to vote/award points!


The problem

After upgrading from Splunk DB Connect V2 to Splunk DB Connect V3 I was disappointed to find that some of the previously useful REST API's I was using to programatically create new connections/data inputs were no longer functional.

After logging a support ticket I was advised these were never supported, and therefore my request is now an "enhancement".

After quite a bit of work I did find a way to automate the DB Connect V3, it is likely unsupported by the official application but it works and provides a nice workaround until they officially support automation via REST API or similar.

Solution

Please see the answer below, comments/better solutions are welcome!

Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

SplunkTrust
SplunkTrust

The solution I have managed to use for connections/database inputs is below, improved solutions are welcome. If you are using DB Connect V2 you might want to try the REST API solution I found https://answers.splunk.com/answers/452618/how-do-i-use-the-restful-webservices-to-setup-new.html#ans... note that this solution does not work for DB Connect V3!

EDIT: 2018-03-20 updated to include comments from ehudb and gsrivastava regarding a new REST endpoint in DB Connect 3.x


Identities

Quoting the answer in this thread from ehudb, use the REST endpoint:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/identities

Example

curl -k -X POST -u admin:changeit https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/identities -d "{\"name\":\"myuser\",\"username\":\"myuser\",\"password\":\"mypassword\"}"

Connections

The workaround solution of editing the dbconnections.conf file and then trigger a get request to:
https://localhost:8089/servicesNS/nobody/splunk
appdbconnect/configs/conf-dbconnections/reload

Is no longer required, as per the comments by gsrivastava you can use:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/connections


Database Inputs

Quoting/paraphrasing the comments from gsrivastava, the URL of:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/inputs

Can be used to create DB connect inputs

JSON data is required, for example:

{
"name": "ABCD",
"query": "select from ABCD",
"interval": "17 ",
"index": "test",
"mode": "rising",
"connection": "abcd",
"rising_column_index": 1,
"timestamp_column_index": 1,
"timestampType": "dbColumn",
"sourcetype": "abcd",
"checkpoint": {"value":"2018-03-22 00:00:00.000","appVersion":"3.1.1","columnType":93,"timestamp":"2018-03-22T11:06:11.000+05:30"}
}

View solution in original post

Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

Communicator

Hi @garethatiag ,

I also wanted to automate the creation of DB Connection in DBX3 and I tried the below but it didnt seem to work:

1) Updated via CLI identities.conf and dbconnections.conf with the new DB details
2) Did a reload on the host
curl -k -u admin:password https://myhost:8089/servicesNS/nobody/splunk
appdbconnect/configs/conf-dbconnections/reload

Is there anything I'm missing out? Could you please guide..

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

SplunkTrust
SplunkTrust

This looks a bit different to what I have done:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload

And:
https://localhost:8089/servicesNS/nobody/search/admin/server/_reload

Can you try those URL's? In particular the second one reloads the task server which should force it to reload the connections...

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

Communicator

Hi @garethatiag ,

Did you create DB coonection and Identity via CLI initially and then did a reload via Splunk RESTAPI? What was your approach?

Im only creating a DB connection for DB Data Query and I do not want to create DB Outputs or inputs so reload inputs URL is not needed for me I think:

However I did try both the URLs that you suggested , reload task-server and reload inputs but it didnt work 😞
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/inputs/_reload

I see the connection in UI but it gives me an error as seen in screenshot:

The splunkappdbconnectserver.log shows below error:

2017-08-03 08:33:46.048 -0400 [dw-49 - GET /api/connections/SPOTFIRE60_DEV/status] ERROR io.dropwizard.jersey.errors.LoggingExceptionMapper - Error handling a request: 94204c89a78800be
java.lang.IllegalArgumentException: Illegal base64 character 5f
at java.util.Base64$Decoder.decode0(Base64.java:714)
at java.util.Base64$Decoder.decode(Base64.java:526)
at java.util.Base64$Decoder.decode(Base64.java:549)
at com.splunk.dbx.crypto.AESUtils.decrypt(AESUtils.java:75)
alt text

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

SplunkTrust
SplunkTrust

If the connection does not work in the UI then the REST API will not help here, I would suggest trying to get it working manually first...

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

Communicator

Hi @garethatiag ,

I noticed that the password is not getting encrypted even after Splunk restart was the root cause.
How does the password get encrypted if restart is not the solution?

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

SplunkTrust
SplunkTrust

I'm using the URL:
https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/

And posting in the values for name/username/password and that takes care of any password encryption for me...

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

Communicator

Hi @garethatiag ,

This URL doesnt update password/username for DB Connect V3. I assume it works only for DBX V2:

https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/identities/

0 Karma
Highlighted

Re: Splunk DB Connect V3 - Automated / Programmatic creation of connections and inputs

SplunkTrust
SplunkTrust

@saranyafmr Unfortunately you are correct, I think I used that url for DBX 2 and now when I re-test with DBX 3 I cannot get that to change, I tested:
/servicesNS/nobody/splunk
appdbconnect/properties/identities
And also:
/servicesNS/nobody/splunkappdbconnect/dbconnect/identities/

And neither work...perhaps an enhancement request to the DB connect team?
It is frustrating that it cannot be automated...

0 Karma