All Apps and Add-ons
Highlighted

DBConnect migration from 2.4 to 3.1 troubleshooting

New Member

Hello,

I am working on a Splunk Entreprise infrastructure and I had to migrate DBConnect from 2.4 to 3.1 on a single instance recently, and I had a lot of trouble to make it work the intended way.

I spent many hours trying to figure out what was wrong and as I saw no documentation for troubleshooting the migration, here is something to help you out.

Installing the app :

The documentation states that you should install the new app then launch the new migration script. In my case it did not work to unzip the new app on top of the old one on the search head, I had a lot of missing changes : 8 changes proposed by the script against 600 with the intended way. The app also didn’t behaved the way it should have. Moving the app directory to a different name then unzipping the new app then launching the migration script also did not work.

I had success with simply updating from the web interface and then launching the script.

Launching the script

The script should be ran with the port where the management port of the component where DBConnect is installed is. Script runs by default on port 8089, so in my case I was trying to run it on the Deployment server management port. I received these errors :

File "./appmigration.py", line 930, in

if "shc
deployer" in client.Entity(service, "server/roles").content.rolelist:
File "/opt/splunk/search
head/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/client.py", line 872, in init
File "/opt/splunk/searchhead/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/client.py", line 1011, in refresh
File "/opt/splunk/search
head/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/client.py", line 981, in get
File "/opt/splunk/searchhead/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/client.py", line 738, in get
File "/opt/splunk/search
head/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/binding.py", line 286, in wrapper
File "/opt/splunk/searchhead/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/binding.py", line 68, in newf
File "/opt/splunk/searchhead/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/binding.py", line 660, in get
File "/opt/splunk/search
head/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/binding.py", line 1150, in get
File "/opt/splunk/searchhead/etc/apps/splunkappdbconnect/bin/splunksdk-1.5.0-py2.7.egg/splunklib/binding.py", line 1205, in request
splunklib.binding.HTTPError: HTTP 404 Not Found -- Application does not exist: splunk
appdbconnect

Changing the port in the command with -port XXXX option worked for me.

HTTP Event Collector

Apparently DBConnect 3 uses the HTTP Event Collector to write into the indexes, at least in my case. It uses port 8088. If that port is occupied, it might seems like everything works, but nothing is written to the indexes. You might encounter these errors in the dbx logs :

2018-06-26 12:01:58.966 +0200 [QuartzSchedulerWorker-8] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unabletowritebatch
java.io.IOException: HTTP Error 401: Unauthorized
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89)
at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36)
at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:79)
at org.easybatch.extensions.quartz.Job.execute(Job.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)


2018-06-26 12:44:16,442 ERROR [5b3219006c7f6ef435a910] config:138 - [HTTP 401] Client is not authenticated
Traceback (most recent call last):
File "/opt/splunk/searchhead/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/config.py", line 136, in getServerZoneInfo
return times.getServerZoneinfo()
File "/opt/splunk/search
head/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/times.py", line 158, in getServerZoneinfo
serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser/tz')
File "/opt/splunk/searchhead/lib/python2.7/site-packages/splunk/rest/init_.py", line 530, in simpleRequest
raise splunk.AuthenticationFailed
AuthenticationFailed: [HTTP 401] Client is not authenticated

Port was occupied in my case so I changed the HTTP Event Port, and restarted Splunk, it worked after that.

I hope that will prove useful for some of you 🙂 !

0 Karma