Cisco eStreamer eNcore Add-on for Splunk: eNcore process not starting

[danicarmelo]

I have a host were TA-eStreamer is deployed, it was working fine last 2018 but it is now not running. This is the estreamer.log when it was working then stopped until the time I tried to start splencore.sh.

2018-11-22 11:20:50,027 Monitor INFO Running. 23229500 handled; average rate 45.3 ev/sec;
2018-11-22 11:23:06,795 Monitor INFO Running. 23230900 handled; average rate 45.29 ev/sec;
2018-11-22 11:23:11,190 Service INFO Splunk is not running.
2018-11-22 11:23:11,191 Service INFO Stopping
2018-11-22 11:23:11,691 Controller INFO Stopping...
2018-11-22 11:23:17,300 SubscriberParser INFO Stop message received
2018-11-22 11:23:27,808 SubscriberParser INFO Exiting
2018-11-22 11:23:27,829 Controller INFO Process 22262 (Process-1) exit code: 0
2018-11-22 11:23:27,835 Decorator INFO Stop message received
2018-11-22 11:23:27,840 Decorator INFO Error state. Clearing queue
2018-11-22 11:23:27,840 Cache INFO Saving cache to $SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/
2018-11-22 11:23:34,042 Decorator INFO Exiting
2018-11-22 11:23:34,154 Controller INFO Process 22263 (Process-2) exit code: 0
2018-11-22 11:23:34,155 Transformer INFO Stop message received
2018-11-22 11:23:34,160 Transformer INFO Error state. Clearing queue
2018-11-22 11:23:34,160 Transformer INFO Exiting
2018-11-22 11:23:34,160 Controller INFO Process 22264 (Process-3) exit code: 0
2018-11-22 11:23:34,161 Writer INFO Stop message received
2018-11-22 11:23:34,166 Writer INFO Error state. Clearing queue
2018-11-22 11:23:34,166 Writer INFO Exiting
2018-11-22 11:23:34,166 Controller INFO Process 22266 (Process-4) exit code: 0
2018-11-22 11:23:34,166 Monitor INFO Stopping Monitor.
2018-11-22 11:23:34,331 Controller INFO Goodbye
2019-10-30 20:07:59,466 Controller INFO eNcore version: 3.5.3

As you can see from the logs that splunk is not running when estreamer logs stopped that time.

But I've verified before and after I've started splencore.sh that splunk is running, but I still see the same message that splunk is not running.

2019-10-31 15:44:39,776 Decorator INFO Starting process.
2019-10-31 15:44:39,777 Transformer INFO Starting process.
2019-10-31 15:44:39,777 Monitor INFO Starting Monitor.
2019-10-31 15:44:39,777 Writer INFO Starting process.
2019-10-31 15:44:39,793 Service INFO Splunk is not running.
2019-10-31 15:44:39,794 Service INFO Stopping

estreamer.logs doesnt really show me why its failing to start.

[douglashurd]

Please update to the latest version of the TA.
https://splunkbase.splunk.com/app/3662/

If you still have the problem just copy / paste new log data in this forum and we'll make a few suggestions.

[danicarmelo]

Hi @douglashurd
I have upgraded to the latest version but I am encountering this error message when i am starting encore:

2019-11-15 21:59:36,939 Diagnostics ERROR The FMC eStreamer server has closed the connection. There are a number of possible causes which may show above in the error log.\n\nIf you see no errors then this could be that:\n * the server is shutting down\n * there has been a client authentication failure (please check that your outbound IP address matches that associated with your certificate - note that if your device is subject to NAT then the certificate IP must match the upstream NAT IP)\n * there is a problem with the server. If you are running FMC v6.0, you may need to install "Sourcefire 3D Defense Center S3 Hotfix AZ 6.1.0.3-1"\n
2019-11-15 21:59:36,940 Controller ERROR ConnectionClosedException: Connection closed\nTraceback (most recent call last):\n File "$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/estreamer/controller.py", line 244, in start\n diagnostics.execute()\n File "$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/estreamer/diagnostics.py", line 96, in execute\n response = connection.response()\n File "$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/estreamer/connection.py", line 181, in response\n dataBuffer = self.__read( 8 )\n File "$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/estreamer/connection.py", line 158, in __read\n raise estreamer.ConnectionClosedException('Connection closed')\nConnectionClosedException: Connection closed\n
2019-11-15 21:59:36,940 Controller INFO Stopping...
2019-11-15 21:59:36,940 Monitor INFO Stopping Monitor.
2019-11-15 21:59:36,941 Controller INFO Goodbye

[vinz2020]

Hi

I am having the same issue with the new app 3.6.8
https://splunkbase.splunk.com/app/3662/

and FMC v6.4.0.7

I can collect the logs a few minutes (cisco:estreamer:data) and then i received
"Process subscriberParser is dead"

any idea ?
thanks a lot

[vik_splunk]

Hi @vinz2020  ,

Did you ever manage to resolve this? We are running into the same issue now

We use the app ver. 4.6.0 on Splunk 8.1.3 with an FMC version of 6.6.0 and are encountering the same issue

[_joe]

It would seem 6.4.0 was released with a couple of bugs. My instance just failed 3-4 days after install. Upgrade to 6.4.2.

[vinz2020]

Yes I fixed it ... but unfortunately I can't remember how 😕

Now I am running app 4.6, Splunk 8.1.3 and FMC 6.5

[vik_splunk]

Thanks. @vinz2020  Any thing that you can recollect and provide inputs will be highly appreciated.

[computermathguy]

Recently we upgraded FMC from 6.x to 7.x and noticed no data was being streamed into the /opt/splunk/etc/apps/TA-eStreamer/bin/encore/data/splunk directory.  We then started getting a firewall error when testing the connection..

Does anyone know if FMC 7.x is compatible with the TA-eStreamer add-on? 

./splencore.sh test
Diagnostics ERROR [no message or attrs]: Could not connect to eStreamer Server at all. Are you sure the host and port are correct? If so then perhaps it is a firewall issue.

[vikesh05]

For me, it turned out to be an incorrect FMC IP. Post proper IP configuration it worked 
Splunk - 9.2.0.1
eStreamer - 5.2.9