DB Connect (dbx) was working fine for 2 weeks now. Without changing anything to the settings of the app or Splunk itself, it happened (out of the blue) that accessing the DB Connect app in Splunk caused the splunkweb to crash. The web interface isn't accessible anymore. I have to kill the splunkweb process in order to restart the splunkd and splunkweb processes. It's reproducible. Here is a part of the web_service.log right before the crash:
2013-04-24 10:57:01,348 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: use_future_expires (bool): True
2013-04-24 10:57:01,348 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: userRegistrationURL (str): https://www.splunk.com/index.php/pre_reg?destination=prod_reg
2013-04-24 10:57:01,349 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: version_label (str): 5.0
2013-04-24 10:57:01,349 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: version_number (str): 5.0
2013-04-24 10:57:01,350 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Bus STARTING
2013-04-24 10:57:01,361 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2013-04-24 10:57:01,469 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Serving on 0.0.0.0:8000
2013-04-24 10:57:01,470 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Bus STARTED
2013-04-24 10:59:19,758 INFO [51779ee7bf1d8e2d0] root:134 - ENGINE: Started monitor thread 'Monitor'.
2013-04-24 11:04:10,586 INFO [5177a00a8c1d8e290] account:240 - user=username action=login status=success session=0c4cd2615e26d1fb8252bb1c2d8572f9 reason=user-initiated useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0" clientip=192.168.40.114
2013-04-24 11:04:10,615 INFO [5177a00a9c1cc6250] cached:77 - memoized decorator used on functionwith non hashable arguments
2013-04-24 11:04:11,048 INFO [5177a00b0b1cc6090] cached:77 - memoized decorator used on functionwith non hashable arguments
2013-04-24 11:04:11,452 INFO [5177a00b0b1cc6090] root:134 - ENGINE: Started monitor thread 'Monitor'.
2013-04-24 11:04:11,717 INFO [5177a00b0b1cc6090] view:1073 - PERF - viewTime=0.3086s templateTime=0.3599s
2013-04-24 11:04:12,898 INFO [5177a00ce42160390] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0, platform=Win32, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (Windows), language=de-DE, oscpu=Windows NT 6.1; WOW64, product=Gecko, productSub=20100101, buildID=20130307023931, doNotTrack=unspecified, mozLockOrientation=function mozLockOrientation() {
[native code]
}, mozUnlockOrientation=function mozUnlockOrientation() {
[native code]
}, availWidth=1366, availHeight=706, width=1366, height=768, colorDepth=24, pixelDepth=24, top=0, left=0, availTop=0, availLeft=0, mozOrientation=landscape-primary, onmozorientationchange=null, addEventListener=function addEventListener() {
[native code]
}, removeEventListener=function removeEventListener() {
[native code]
}, dispatchEvent=function dispatchEvent() {
[native code]
}, documentURL=https://dessplunk:8000/en-US/app/search/dashboard_live, documentReferrer=https://dessplunk:8000/en-US/account/login?return_to=%2Fen-US%2F, flash=11.6.602, Splunk.Session.START_EVENT fired @Wed Apr 24 2013 11:04:12 GMT+0200
2013-04-24 11:04:16,672 INFO [5177a010ab2386f50] cached:77 - memoized decorator used on functionwith non hashable arguments
2013-04-24 11:04:16,908 INFO [5177a010e72124090] cached:77 - memoized decorator used on functionwith non hashable arguments
2013-04-24 11:04:17,413 INFO [5177a010e72124090] view:1073 - PERF - viewTime=0.2569s templateTime=0.2476s
2013-04-24 11:04:18,627 INFO [5177a0129f2160590] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0, platform=Win32, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (Windows), language=de-DE, oscpu=Windows NT 6.1; WOW64, product=Gecko, productSub=20100101, buildID=20130307023931, doNotTrack=unspecified, mozLockOrientation=function mozLockOrientation() {
[native code]
}, mozUnlockOrientation=function mozUnlockOrientation() {
[native code]
}, availWidth=1366, availHeight=706, width=1366, height=768, colorDepth=24, pixelDepth=24, top=0, left=0, availTop=0, availLeft=0, mozOrientation=landscape-primary, onmozorientationchange=null, addEventListener=function addEventListener() {
[native code]
}, removeEventListener=function removeEventListener() {
[native code]
}, dispatchEvent=function dispatchEvent() {
[native code]
}, documentURL=https://dessplunk:8000/en-US/app/dbx/home, documentReferrer=https://dessplunk:8000/en-US/app/search/dashboard_live, flash=11.6.602, Splunk.Session.START_EVENT fired @Wed Apr 24 2013 11:04:17 GMT+0200
2013-04-24 11:04:22,259 INFO [5177a0160c21d4310] cached:77 - memoized decorator used on functionwith non hashable arguments
2013-04-24 11:04:22,906 INFO [5177a0160c21d4310] cached:77 - memoized decorator used on functionwith non hashable arguments
Fixed it.
"ps x | grep splunk" showed me, that my db queries were still running after i stopped splunkd. killing the process "/usr/lib64/jvm/jre-1.6.0-ibm/bin/java" solved it. now i'm able to use db connect in splunk again.
Fixed it.
"ps x | grep splunk" showed me, that my db queries were still running after i stopped splunkd. killing the process "/usr/lib64/jvm/jre-1.6.0-ibm/bin/java" solved it. now i'm able to use db connect in splunk again.