version 4.2.3
Once or twice in a 24hr period we get a gray notification on the Splunk dashboard with regards to rtsearch timing out or being terminated.
rt_admin_admin_search_Q0hBUlQgSUlTIFdlYnNoaXRzIGJ5IEhUVFAgc3RhdHVz_rt_1319177401.105
This causes the real time dashboard to stop working. In our NOC this isn't very useful.
We run the Splunk dash in a Chrome browser which connects over a VPN to our DC. Could this be a connection fault causing these breaks? No other monitoring tools have the same issue.
Also, the dashboard is running several searches in realtime, saving 1hr of historic data each.
Here are our Splunkd.log for the time of which the last RTSEARCH CONNECTION TERMINATED in the dashboard.
Please let me know if any other information is required. It has been so far the only issue we have had with Splunk.
10-21-2011 09:05:23.228 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND blocked index::main source::d:\\apps\\webknight\\* ]', active_streams = 8
10-21-2011 09:05:26.582 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 7
10-21-2011 09:05:29.422 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 6
10-21-2011 09:22:25.131 +0000 INFO WatchedFile - Will begin reading at offset=24999901 for file='D:\APPS\Splunk\var\log\splunk\audit.log.1'.
10-21-2011 09:22:26.379 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\APPS\Splunk\var\log\splunk\audit.log'.
10-21-2011 09:22:26.379 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='D:\APPS\Splunk\var\log\splunk\audit.log'.
10-21-2011 09:43:01.768 +0000 WARN DateParserVerbose - Failed to parse timestamp for event.
Context="source::D:\IISWEB\Logs\Logfiles\W3SVC4\u_ex111021.log|host::CODSCL01|iisw3c|remoteport::22081" Text="#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-usern..."
10-21-2011 10:05:36.802 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 5
10-21-2011 10:05:36.802 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 4
10-21-2011 10:05:37.738 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND 40* index::main sourcetype::iisw3c ]', active_streams = 3
10-21-2011 10:05:37.816 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 2
10-21-2011 10:05:40.109 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::webknight ]', active_streams = 1
10-21-2011 10:05:46.599 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND 50* index::main sourcetype::iisw3c ]', active_streams = 0
10-21-2011 10:08:48.648 +0000 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::D:\IISWEB\Logs\Logfiles\W3SVC4\u_ex111021.log|host::CODSCL02|iisw3c|remoteport::22209"
This is what generates the Pie Chart.
sourcetype="iisw3c" | geoip c_ip | dedup c_ip | top c_ip_country_name
Splunkweb errors.
06:55:56.127 2011-10-21 07:55:56,127 ERROR [4ea1258c1c8c46908] utility:59 - name=javascript, class=Splunk.Error, lineNumber=3958, message=getConfigValue - SERVER_ZONEINFO not set, no default provided, fileName=http://x/en-GB/static/@105575/js/common.min.js
21/10/2011 06:54:25.969 2011-10-21 07:54:25,969 ERROR [4ea12531f889242e8] utility:59 - name=javascript, class=Splunk.Error, lineNumber=884, message=Unspecified error., fileName=http://x/en-GB/static/@105575/js/common.min.js
20/10/2011 16:29:04.105 2011-10-20 17:29:04,105 ERROR [4ea05a5fc77886cc0] admin:944 - uiHelper processValueEdit operator failed for endpoint_path=saved/searches/PieChart IP's by Geographical Location elementName=spl-ctrl_summary_index: argument of type 'NoneType' is not iterable
20/10/2011 16:29:04.105 2011-10-20 17:29:04,105 ERROR [4ea05a5fc77886cc0] admin:944 - uiHelper processValueEdit operator failed for endpoint_path=saved/searches/PieChart IP's by Geographical Location elementName=spl-ctrl_script_enable: argument of type 'NoneType' is not iterable
20/10/2011 16:29:04.105 2011-10-20 17:29:04,105 ERROR [4ea05a5fc77886cc0] admin:944 - uiHelper processValueEdit operator failed for endpoint_path=saved/searches/PieChart IP's by Geographical Location elementName=spl-ctrl_rss_enable: argument of type 'NoneType' is not iterable