DB Connect causes Splunkweb to crash

[markovic]

DB Connect (dbx) was working fine for 2 weeks now. Without changing anything to the settings of the app or Splunk itself, it happened (out of the blue) that accessing the DB Connect app in Splunk caused the splunkweb to crash. The web interface isn't accessible anymore. I have to kill the splunkweb process in order to restart the splunkd and splunkweb processes. It's reproducible. Here is a part of the web_service.log right before the crash:

2013-04-24 10:57:01,348 INFO   [51779e5c9fdc4ed0] root:535 - CONFIG: use_future_expires (bool): True

2013-04-24 10:57:01,348 INFO    [51779e5c9fdc4ed0] root:535 - CONFIG: userRegistrationURL (str): https://www.splunk.com/index.php/pre_reg?destination=prod_reg

2013-04-24 10:57:01,349 INFO    [51779e5c9fdc4ed0] root:535 - CONFIG: version_label (str): 5.0

2013-04-24 10:57:01,349 INFO    [51779e5c9fdc4ed0] root:535 - CONFIG: version_number (str): 5.0

2013-04-24 10:57:01,350 INFO    [51779e5c9fdc4ed0] root:134 - ENGINE: Bus STARTING

2013-04-24 10:57:01,361 INFO    [51779e5c9fdc4ed0] root:134 - ENGINE: Started monitor thread '_TimeoutMonitor'.

2013-04-24 10:57:01,469 INFO    [51779e5c9fdc4ed0] root:134 - ENGINE: Serving on 0.0.0.0:8000

2013-04-24 10:57:01,470 INFO    [51779e5c9fdc4ed0] root:134 - ENGINE: Bus STARTED

2013-04-24 10:59:19,758 INFO    [51779ee7bf1d8e2d0] root:134 - ENGINE: Started monitor thread 'Monitor'.

2013-04-24 11:04:10,586 INFO    [5177a00a8c1d8e290] account:240 - user=username action=login status=success session=0c4cd2615e26d1fb8252bb1c2d8572f9 reason=user-initiated useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0" clientip=192.168.40.114

2013-04-24 11:04:10,615 INFO    [5177a00a9c1cc6250] cached:77 - memoized decorator used on function  with non hashable arguments

2013-04-24 11:04:11,048 INFO    [5177a00b0b1cc6090] cached:77 - memoized decorator used on function  with non hashable arguments

2013-04-24 11:04:11,452 INFO    [5177a00b0b1cc6090] root:134 - ENGINE: Started monitor thread 'Monitor'.

2013-04-24 11:04:11,717 INFO    [5177a00b0b1cc6090] view:1073 - PERF - viewTime=0.3086s templateTime=0.3599s

2013-04-24 11:04:12,898 INFO    [5177a00ce42160390] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0, platform=Win32, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (Windows), language=de-DE, oscpu=Windows NT 6.1; WOW64, product=Gecko, productSub=20100101, buildID=20130307023931, doNotTrack=unspecified, mozLockOrientation=function mozLockOrientation() {

    [native code]

}, mozUnlockOrientation=function mozUnlockOrientation() {

    [native code]

}, availWidth=1366, availHeight=706, width=1366, height=768, colorDepth=24, pixelDepth=24, top=0, left=0, availTop=0, availLeft=0, mozOrientation=landscape-primary, onmozorientationchange=null, addEventListener=function addEventListener() {

    [native code]

}, removeEventListener=function removeEventListener() {

    [native code]

}, dispatchEvent=function dispatchEvent() {

    [native code]

}, documentURL=https://dessplunk:8000/en-US/app/search/dashboard_live, documentReferrer=https://dessplunk:8000/en-US/account/login?return_to=%2Fen-US%2F, flash=11.6.602, Splunk.Session.START_EVENT fired @Wed Apr 24 2013 11:04:12 GMT+0200

2013-04-24 11:04:16,672 INFO    [5177a010ab2386f50] cached:77 - memoized decorator used on function  with non hashable arguments

2013-04-24 11:04:16,908 INFO    [5177a010e72124090] cached:77 - memoized decorator used on function  with non hashable arguments

2013-04-24 11:04:17,413 INFO    [5177a010e72124090] view:1073 - PERF - viewTime=0.2569s templateTime=0.2476s

2013-04-24 11:04:18,627 INFO    [5177a0129f2160590] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0, platform=Win32, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (Windows), language=de-DE, oscpu=Windows NT 6.1; WOW64, product=Gecko, productSub=20100101, buildID=20130307023931, doNotTrack=unspecified, mozLockOrientation=function mozLockOrientation() {

    [native code]

}, mozUnlockOrientation=function mozUnlockOrientation() {

    [native code]

}, availWidth=1366, availHeight=706, width=1366, height=768, colorDepth=24, pixelDepth=24, top=0, left=0, availTop=0, availLeft=0, mozOrientation=landscape-primary, onmozorientationchange=null, addEventListener=function addEventListener() {

    [native code]

}, removeEventListener=function removeEventListener() {

    [native code]

}, dispatchEvent=function dispatchEvent() {

    [native code]

}, documentURL=https://dessplunk:8000/en-US/app/dbx/home, documentReferrer=https://dessplunk:8000/en-US/app/search/dashboard_live, flash=11.6.602, Splunk.Session.START_EVENT fired @Wed Apr 24 2013 11:04:17 GMT+0200

2013-04-24 11:04:22,259 INFO    [5177a0160c21d4310] cached:77 - memoized decorator used on function  with non hashable arguments

2013-04-24 11:04:22,906 INFO    [5177a0160c21d4310] cached:77 - memoized decorator used on function  with non hashable arguments

[markovic · Accepted solution]

Fixed it.
"ps x | grep splunk" showed me, that my db queries were still running after i stopped splunkd. killing the process "/usr/lib64/jvm/jre-1.6.0-ibm/bin/java" solved it. now i'm able to use db connect in splunk again.