splunk web service down?

[khyoung7410]

Hi

I have a two indexer and one search header.

Licence is 200GB. splunk version is 4.3.2. OS is RedHat 5.5 Ent. CPU is 2.4GH 2 quard core. MEM is 8GB

search header of splunk web service down. only splunk web service down. splunkd service alive.

Is it because time? only search header and ntpserver linkage.

Do you have splunk web service down known issue?

Thank you.

-- web_service.log
2012-06-21 01:53:52,846 INFO    [4fe20020d712a00f50] utility:63 - name=javascript, class=Splunk.Session, appCodeName=Mozilla, appName=Microsoft Internet Explorer, appMinorVersion=0, cpuClass=x86, platform=Win32, systemLanguage=ko, userLanguage=ko, appVersion=4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), userAgent=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), height=1080, bufferDepth=0, deviceXDPI=96, logicalYDPI=96, deviceYDPI=96, availHeight=1080, logicalXDPI=96, systemXDPI=96, fontSmoothingEnabled=true, systemYDPI=96, colorDepth=32, width=1920, availWidth=1920, updateInterval=0, documentURL=http://10.140.1.50:8000/ko-KR/manager/search/saved/searches/%ED%95%98%EB%82%98%20%EC%BA%90%ED%94%BC%ED%83%88%20APC?action=edit&ns=search&uri=%2FservicesNS%2Fadmin%2Fsearch%2Fsaved%2Fsearches%2F%25ED%2595%2598%25EB%2582%2598%2520%25EC%25BA%2590%25ED%2594%25BC%25ED%2583%2588%2520APC, documentReferrer=http://10.140.1.50:8000/ko-KR/manager/search/saved/searches, flash=11.2.202, Splunk.Session.START_EVENT fired @Thu Jun 21 01:54:09 UTC+0900 2012
2012-06-21 01:53:56,215 INFO    [4fe200240e2aaaca180d50] cached:77 - memoized decorator used on function <function getEntities at 0x1181e398> with non hashable arguments
2012-06-21 01:53:56,435 INFO    [4fe200240e2aaaca180d50] cached:77 - memoized decorator used on function <function getEntities at 0x1181e398> with non hashable arguments
2012-06-21 01:53:57,738 INFO    [4fe20025bb13a79750] utility:63 - name=javascript, class=Splunk.Session, appCodeName=Mozilla, appName=Microsoft Internet Explorer, appMinorVersion=0, cpuClass=x86, platform=Win32, systemLanguage=ko, userLanguage=ko, appVersion=4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), userAgent=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), height=1080, bufferDepth=0, deviceXDPI=96, logicalYDPI=96, deviceYDPI=96, availHeight=1080, logicalXDPI=96, systemXDPI=96, fontSmoothingEnabled=true, systemYDPI=96, colorDepth=32, width=1920, availWidth=1920, updateInterval=0, documentURL=http://10.140.1.50:8000/ko-KR/manager/search/saved/searches?msgid=454110.617617482516&ns=search&redirecting=true, documentReferrer=, flash=11.2.202, Splunk.Session.START_EVENT fired @Thu Jun 21 01:54:14 UTC+0900 2012
2012-06-21 01:53:58,618 INFO    [4fe200269d2aaaca180650] cached:77 - memoized decorator used on function <function getEntities at 0x1181e398> with non hashable arguments
2012-06-21 01:53:58,741 INFO    [4fe20026bd13a79d10] cached:77 - memoized decorator used on function <function getEntities at 0x1181e398> with non hashable arguments
2012-06-21 01:53:58,930 INFO    [4fe20026bd13a79d10] view:1059 - PERF - viewTime=0.1424s templateTime=0.0468s
2012-06-21 01:54:00,043 INFO    [4fe200280a2aaab47dfa90] utility:63 - name=javascript, class=Splunk.Session, appCodeName=Mozilla, appName=Microsoft Internet Explorer, appMinorVersion=0, cpuClass=x86, platform=Win32, systemLanguage=ko, userLanguage=ko, appVersion=4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), userAgent=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), height=1080, bufferDepth=0, deviceXDPI=96, logicalYDPI=96, deviceYDPI=96, availHeight=1080, logicalXDPI=96, systemXDPI=96, fontSmoothingEnabled=true, systemYDPI=96, colorDepth=32, width=1920, availWidth=1920, updateInterval=0, documentURL=http://10.140.1.50:8000/ko-KR/app/search/dashboard_live, documentReferrer=http://10.140.1.50:8000/ko-KR/manager/search/saved/searches?msgid=454110.617617482516&ns=search&redirecting=true, flash=11.2.202, Splunk.Session.START_EVENT fired @Thu Jun 21 01:54:17 UTC+0900 2012
2012-06-21 01:54:03,639 INFO    [4fe2002b9f139dae90] view:537 - loading saved search "?섎굹 罹먰뵾??APC" into view "flashtimeline"
2012-06-21 01:54:03,647 INFO    [4fe2002ba4139da590] cached:77 - memoized decorator used on function <function getEntities at 0x1181e398> with non hashable arguments
2012-06-21 01:54:03,950 INFO    [4fe2002ba4139da590] view:1059 - PERF - viewTime=0.1791s templateTime=0.1244s
2012-06-21 01:54:05,696 INFO    [4fe2002db12aaaac0ea210] utility:63 - name=javascript, class=Splunk.Session, appCodeName=Mozilla, appName=Microsoft Internet Explorer, appMinorVersion=0, cpuClass=x86, platform=Win32, systemLanguage=ko, userLanguage=ko, appVersion=4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), userAgent=Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2), height=1080, bufferDepth=0, deviceXDPI=96, logicalYDPI=96, deviceYDPI=96, availHeight=1080, logicalXDPI=96, systemXDPI=96, fontSmoothingEnabled=true, systemYDPI=96, colorDepth=32, width=1920, availWidth=1920, updateInterval=0, documentURL=http://10.140.1.50:8000/ko-KR/app/search/flashtimeline?s=%ED%95%98%EB%82%98%20%EC%BA%90%ED%94%BC%ED%83%88%20APC, documentReferrer=http://10.140.1.50:8000/ko-KR/app/search/dashboard_live, flash=11.2.202, Splunk.Session.START_EVENT fired @Thu Jun 21 01:54:23 UTC+0900 2012
2012-06-21 01:54:23,454 INFO    [4fe2003f732aaac7575e50] <string>:59 - creating excel export for user admin from search id 1340211248.13715
2012-06-21 01:54:24,508 INFO    [4fe2003f732aaac7575e50] wb_client:68 - request completed successfully, parsed 4 results
2012-06-21 01:54:24,508 INFO    [4fe2003f732aaac7575e50] <string>:125 - streaming excel spreadsheet to client
2012-06-21 01:54:48,696 INFO    [4fe20058b12aaab55f62d0] <string>:59 - creating excel export for user admin from search id 1340210762.13682
2012-06-21 01:56:48,862 INFO    [4fe200d0db12bce610] <string>:59 - creating excel export for user admin from search id 1340210762.13682
2012-06-21 01:57:01,381 INFO    [4fe200d0db12bce610] wb_client:68 - request completed successfully, parsed 9 results
2012-06-21 02:08:08,671 INFO    [4fe2037881efaf10] __init__:160 - Using default logging config file: /opt/splunk/etc/log.cfg
2012-06-21 02:08:08,671 INFO    [4fe2037881efaf10] __init__:176 - Setting logger=splunk level=INFO
2012-06-21 02:08:08,672 INFO    [4fe2037881efaf10] __init__:176 - Setting logger=splunk.appserver level=INFO
2012-06-21 02:08:08,672 INFO    [4fe2037881efaf10] __init__:176 - Setting logger=splunk.appserver.controllers level=INFO
2012-06-21 02:08:08,672 INFO    [4fe2037881efaf10] __init__:176 - Setting logger=splunk.appserver.lib level=WARN
2012-06-21 02:08:09,381 INFO    [4fe2037881efaf10] lists:59 - List controller loaded: EntitiesListGenerator
2012-06-21 02:08:09,381 INFO    [4fe2037881efaf10] lists:65 - Setting lists/entities
2012-06-21 02:08:09,381 INFO    [4fe2037881efaf10] lists:59 - List controller loaded: JobsListGenerator
2012-06-21 02:08:09,382 INFO    [4fe2037881efaf10] lists:65 - Setting lists/jobs

[Drainy]

Ok, just to take it back a few steps. What made you know that splunkweb was down? is this an old system using the default ports and you suddenly found you couldn't access it? Also, what happens when you try to restart it?

[khyoung7410]

question updated with web logs.

[sideview]

splunkd.log won't say much about splunkweb. I'd remove the splunkd.log from your question and replace it with anything interesting you can find at the end of splunk_web_service.log

[rgonzale6]

It's possible your splunkweb has simply crashed. I see nothing in your logs that would indicate why. The time issue is not likely to have been the cause.

Have you tried...

./opt/splunk/bin/splunk start splunkweb

?

[sideview]

Well the admin and power roles inherit some of the base capabilities from the user role, so I think it's quite possible that if you gut core capabilities out of the user role, you could inadvertently get into a state where no user has basic capabilities to look at any views or dashboards in Splunk. Are you sure splunkWeb isn't running. Have you run "splunk status" from the command line?

[AlexMcDuffMille]

Same thing happened to me after I changed the settings for the user role. I was logged in as admin.

[Splunk_U]

Have you got any answer of your question?? I have faced the same issue...

[khyoung7410]

I know splunkweb start.
I want to splunkweb demon down reason.