version 4.2.3
Once or twice in a 24hr period we get a gray notification on the Splunk dashboard with regards to rtsearch timing out or being terminated.
rt_admin_admin_search_Q0hBUlQgSUlTIFdlYnNoaXRzIGJ5IEhUVFAgc3RhdHVz_rt_1319177401.105
This causes the real time dashboard to stop working. In our NOC this isn't very useful.
We run the Splunk dash in a Chrome browser which connects over a VPN to our DC. Could this be a connection fault causing these breaks? No other monitoring tools have the same issue.
Also, the dashboard is running several searches in realtime, saving 1hr of historic data each.
Here are our Splunkd.log for the time of which the last RTSEARCH CONNECTION TERMINATED in the dashboard.
Please let me know if any other information is required. It has been so far the only issue we have had with Splunk.
10-21-2011 09:05:23.228 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND blocked index::main source::d:\\apps\\webknight\\* ]', active_streams = 8
10-21-2011 09:05:26.582 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 7
10-21-2011 09:05:29.422 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 6
10-21-2011 09:22:25.131 +0000 INFO WatchedFile - Will begin reading at offset=24999901 for file='D:\APPS\Splunk\var\log\splunk\audit.log.1'.
10-21-2011 09:22:26.379 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\APPS\Splunk\var\log\splunk\audit.log'.
10-21-2011 09:22:26.379 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='D:\APPS\Splunk\var\log\splunk\audit.log'.
10-21-2011 09:43:01.768 +0000 WARN DateParserVerbose - Failed to parse timestamp for event.
Context="source::D:\IISWEB\Logs\Logfiles\W3SVC4\u_ex111021.log|host::CODSCL01|iisw3c|remoteport::22081" Text="#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-usern..."
10-21-2011 10:05:36.802 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 5
10-21-2011 10:05:36.802 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 4
10-21-2011 10:05:37.738 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND 40* index::main sourcetype::iisw3c ]', active_streams = 3
10-21-2011 10:05:37.816 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::iisw3c ]', active_streams = 2
10-21-2011 10:05:40.109 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND index::main sourcetype::webknight ]', active_streams = 1
10-21-2011 10:05:46.599 +0000 INFO IndexProcessor - rtsearch connection terminated, filter = '[ AND 50* index::main sourcetype::iisw3c ]', active_streams = 0
10-21-2011 10:08:48.648 +0000 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::D:\IISWEB\Logs\Logfiles\W3SVC4\u_ex111021.log|host::CODSCL02|iisw3c|remoteport::22209"
... View more