Monitoring Splunk

DB Connect causes Splunkweb to crash

markovic
Explorer

DB Connect (dbx) was working fine for 2 weeks now. Without changing anything to the settings of the app or Splunk itself, it happened (out of the blue) that accessing the DB Connect app in Splunk caused the splunkweb to crash. The web interface isn't accessible anymore. I have to kill the splunkweb process in order to restart the splunkd and splunkweb processes. It's reproducible. Here is a part of the web_service.log right before the crash:

2013-04-24 10:57:01,348 INFO   [51779e5c9fdc4ed0] root:535 - CONFIG: use_future_expires (bool): True
2013-04-24 10:57:01,348 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: userRegistrationURL (str): https://www.splunk.com/index.php/pre_reg?destination=prod_reg
2013-04-24 10:57:01,349 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: version_label (str): 5.0
2013-04-24 10:57:01,349 INFO [51779e5c9fdc4ed0] root:535 - CONFIG: version_number (str): 5.0
2013-04-24 10:57:01,350 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Bus STARTING
2013-04-24 10:57:01,361 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Started monitor thread '_TimeoutMonitor'.
2013-04-24 10:57:01,469 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Serving on 0.0.0.0:8000
2013-04-24 10:57:01,470 INFO [51779e5c9fdc4ed0] root:134 - ENGINE: Bus STARTED
2013-04-24 10:59:19,758 INFO [51779ee7bf1d8e2d0] root:134 - ENGINE: Started monitor thread 'Monitor'.
2013-04-24 11:04:10,586 INFO [5177a00a8c1d8e290] account:240 - user=username action=login status=success session=0c4cd2615e26d1fb8252bb1c2d8572f9 reason=user-initiated useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0" clientip=192.168.40.114
2013-04-24 11:04:10,615 INFO [5177a00a9c1cc6250] cached:77 - memoized decorator used on function with non hashable arguments
2013-04-24 11:04:11,048 INFO [5177a00b0b1cc6090] cached:77 - memoized decorator used on function with non hashable arguments
2013-04-24 11:04:11,452 INFO [5177a00b0b1cc6090] root:134 - ENGINE: Started monitor thread 'Monitor'.
2013-04-24 11:04:11,717 INFO [5177a00b0b1cc6090] view:1073 - PERF - viewTime=0.3086s templateTime=0.3599s
2013-04-24 11:04:12,898 INFO [5177a00ce42160390] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0, platform=Win32, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (Windows), language=de-DE, oscpu=Windows NT 6.1; WOW64, product=Gecko, productSub=20100101, buildID=20130307023931, doNotTrack=unspecified, mozLockOrientation=function mozLockOrientation() {
[native code]
}, mozUnlockOrientation=function mozUnlockOrientation() {
[native code]
}, availWidth=1366, availHeight=706, width=1366, height=768, colorDepth=24, pixelDepth=24, top=0, left=0, availTop=0, availLeft=0, mozOrientation=landscape-primary, onmozorientationchange=null, addEventListener=function addEventListener() {
[native code]
}, removeEventListener=function removeEventListener() {
[native code]
}, dispatchEvent=function dispatchEvent() {
[native code]
}, documentURL=https://dessplunk:8000/en-US/app/search/dashboard_live, documentReferrer=https://dessplunk:8000/en-US/account/login?return_to=%2Fen-US%2F, flash=11.6.602, Splunk.Session.START_EVENT fired @Wed Apr 24 2013 11:04:12 GMT+0200
2013-04-24 11:04:16,672 INFO [5177a010ab2386f50] cached:77 - memoized decorator used on function with non hashable arguments
2013-04-24 11:04:16,908 INFO [5177a010e72124090] cached:77 - memoized decorator used on function with non hashable arguments
2013-04-24 11:04:17,413 INFO [5177a010e72124090] view:1073 - PERF - viewTime=0.2569s templateTime=0.2476s
2013-04-24 11:04:18,627 INFO [5177a0129f2160590] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0, platform=Win32, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (Windows), language=de-DE, oscpu=Windows NT 6.1; WOW64, product=Gecko, productSub=20100101, buildID=20130307023931, doNotTrack=unspecified, mozLockOrientation=function mozLockOrientation() {
[native code]
}, mozUnlockOrientation=function mozUnlockOrientation() {
[native code]
}, availWidth=1366, availHeight=706, width=1366, height=768, colorDepth=24, pixelDepth=24, top=0, left=0, availTop=0, availLeft=0, mozOrientation=landscape-primary, onmozorientationchange=null, addEventListener=function addEventListener() {
[native code]
}, removeEventListener=function removeEventListener() {
[native code]
}, dispatchEvent=function dispatchEvent() {
[native code]
}, documentURL=https://dessplunk:8000/en-US/app/dbx/home, documentReferrer=https://dessplunk:8000/en-US/app/search/dashboard_live, flash=11.6.602, Splunk.Session.START_EVENT fired @Wed Apr 24 2013 11:04:17 GMT+0200
2013-04-24 11:04:22,259 INFO [5177a0160c21d4310] cached:77 - memoized decorator used on function with non hashable arguments
2013-04-24 11:04:22,906 INFO [5177a0160c21d4310] cached:77 - memoized decorator used on function with non hashable arguments

Tags (2)
0 Karma
1 Solution

markovic
Explorer

Fixed it.
"ps x | grep splunk" showed me, that my db queries were still running after i stopped splunkd. killing the process "/usr/lib64/jvm/jre-1.6.0-ibm/bin/java" solved it. now i'm able to use db connect in splunk again.

View solution in original post

0 Karma

markovic
Explorer

Fixed it.
"ps x | grep splunk" showed me, that my db queries were still running after i stopped splunkd. killing the process "/usr/lib64/jvm/jre-1.6.0-ibm/bin/java" solved it. now i'm able to use db connect in splunk again.

0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...