When running a search which takes longer than a couple of seconds to complete, I suddenly see the following error messages. Unknown sid. The search job “1522944681.58347” was canceled remotely or expired. These messages are displayed approximately one second after the search starts running. If I check the $SPLUNK_HOME/var/run/splunk/dispatch directory, I can see the search artifact getting created, and being reaped right after the search is cancelled. This does not seem to be affecting saved searches which run on a schedule. This is also a standalone Splunk instance running version 6.5.2. What is the cause of this? ... View more

After enabling the Distributed Management Console on an Enterprise Security (ES) search head, searches stop returning results. The following additional behaviors were also experienced. Navigation to the Settings menu in the UI is slow but works. When trying to access the Search UI, Splunk does not respond. Running a search from the command line does not return results. The “Scheduled time” value in “Settings > Searches, reports, and alerts” contains dates from the past. Large gaps in time (more than 24 hours in some cases) were seen in the scheduler.log and splunkd.log files. Why is this happening? ... View more

From can I see, Splunk continues to run but I would like to know what happens to the cold data which meets the criteria to be frozen? Once the frozen directory is made accessible, does Splunk continue to freeze the data, or was it already removed from the index? ... View more