eNcore running but error and no data displayed

[pmeyerson]

I received input on the encore dashboard for a day and then it stopped processing. I see this frequently in the estreamer.log file:

2018-03-27 11:16:35,058 estreamer.client INFO Process handler state is Error.
2018-03-27 11:16:35,058 estreamer.monitor INFO Running. 0 subscribed; 0 handled;
2018-03-27 11:16:35,501 estreamer.client INFO Stopping...
2018-03-27 11:16:36,145 estreamer.subscriber INFO Stop message received
2018-03-27 11:16:36,146 estreamer.subscriber INFO Waiting for messageQueue to clear...
2018-03-27 11:16:36,146 estreamer.subscriber INFO Exiting
2018-03-27 11:16:36,146 estreamer.client INFO Process 19018 (subscriber) exit code: 0
2018-03-27 11:16:36,146 estreamer.handler INFO Stop message received
2018-03-27 11:16:36,146 estreamer.handler INFO Exiting
2018-03-27 11:16:36,147 estreamer.client INFO Process 19017 (handler) exit code: 0
2018-03-27 11:16:36,147 estreamer.monitor INFO Stopping Monitor.
2018-03-27 11:16:36,308 estreamer.client INFO Goodbye
2018-03-27 11:16:37,863 estreamer.client INFO eNcore version: 3.0.0
2018-03-27 11:16:37,864 estreamer.client INFO Python version: 2.7.13 (default, Apr 16 2017, 01:25:17) \n[GCC 5.3.0]
2018-03-27 11:16:37,864 estreamer.client INFO Platform version: Linux-4.4.0-116-generic-x86_64-with-debian-stretch-sid
2018-03-27 11:16:37,864 estreamer.client INFO Starting client (pid=25163).
2018-03-27 11:16:37,864 estreamer.client INFO Sha256: xxxxx
2018-03-27 11:16:37,865 Diagnostics INFO Check certificate
2018-03-27 11:16:37,865 Diagnostics INFO Creating connection
2018-03-27 11:16:37,865 estreamer.connection INFO Connecting to xxxxx:8302
2018-03-27 11:16:37,865 estreamer.connection INFO Using TLS v1.2
2018-03-27 11:16:38,108 Diagnostics INFO Creating request message
2018-03-27 11:16:38,108 Diagnostics INFO Request message=zzzzz
2018-03-27 11:16:38,109 Diagnostics INFO Sending request message
2018-03-27 11:16:38,109 Diagnostics INFO Receiving response message
2018-03-27 11:16:38,130 Diagnostics INFO Response message=zzzzzz
2018-03-27 11:16:38,131 Diagnostics INFO Streaming info response
2018-03-27 11:16:38,131 Diagnostics INFO Connection successful
2018-03-27 11:16:38,132 estreamer.monitor INFO Starting Monitor.
2018-03-27 11:16:38,132 estreamer.metadata.cache INFO Loading cache from /opt/splunk/etc/apps/TA-eStreamer/bin/encore/xxxxx-8302_cache.dat
2018-03-27 11:16:38,133 estreamer.subscriber INFO Starting Subscriber.
2018-03-27 11:16:38,133 estreamer.connection INFO Connecting to xxxxx:8302
2018-03-27 11:16:38,134 estreamer.connection INFO Using TLS v1.2
2018-03-27 11:16:38,135 estreamer.monitor INFO Starting. 0 subscribed; 0 handled;
2018-03-27 11:16:38,417 estreamer.handler ERROR ValueError: insecure string pickle\nTraceback (most recent call last):\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/handler.py", line 116, in start\n self.output = estreamer.outputters.Manager( outputters, self.settings )\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/outputters/manager.py", line 36, in init\n self.cache.load()\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/metadata/cache.py", line 532, in load\n data = cPickle.load( cacheFile )\nValueError: insecure string pickle\n
2018-03-27 11:16:38,418 estreamer.handler INFO Error state. Clearing queue
2018-03-27 11:16:38,619 estreamer.bookmark INFO Opening bookmark file /opt/splunk/etc/apps/TA-eStreamer/bin/encore/xxxxx-8302_bookmark.dat.
2018-03-27 11:16:38,620 estreamer.settings.settings INFO Timestamp: Start = 2 (Bookmark = 1521499872)
2018-03-27 11:16:38,623 estreamer.subscriber INFO EventStreamRequestMessage: 00010002000000085ab03ee048900061
2018-03-27 11:16:38,623 estreamer.bookmark INFO Opening bookmark file /opt/splunk/etc/apps/TA-eStreamer/bin/encore/xxxxx-8302_bookmark.dat.
2018-03-27 11:16:38,623 estreamer.settings.settings INFO Timestamp: Start = 2 (Bookmark = 1521499872)
2018-03-27 11:16:38,623 estreamer.subscriber INFO StreamingRequestMessage: zzzzz

Looking for some troubleshooting assistance.

[pmeyerson]

Oh, thats very helpful to know, thanks. Yeah this is my test splunk server and I blew the disk space at one point, I'll give that a try.

[lakshman239 · Accepted solution]

Did you run out of disk space before seeing these exceptions? The TA seems to show (at least in my case) the above errors after running of disk space, as the data files are getting written to TA-eStreamer/data/encore*.log
Disable/enable of the TA didn't help after clearning the disk space. I had to stop the instance, remove the bookmark, cache, data, pid files under TA-eStreamer/bin/encore and restart the instance to get new files getting generated and processed.

[pmeyerson]

Can you clarify "stop the instance"? You mean the "is enabled?" checkbox in the app setup?

I actually got a save failed error, I'll try to re-install the app this week.

Encountered the following error while trying to update: Splunkd daemon is not responding: (u"Error connecting to /servicesNS/nobody/TA-eStreamer/apps/local/TA-eStreamer/setup: ('The read operation timed out',)",)

[pmeyerson]

on review this was the step that seemed to fix the issue, I think I didn't realize I was having more disk space issues.