Why is Splunk eNcore not receiving any Data?

[fwbagusf]

Hi All,

Currently we noticed that splunk eNcore not receiving any Data, the error log on cisco:estreamer:log said ERROR EncoreException: PID file already exists. So we assumed that there is conflicted eNcore process, so we try to ./splencore.sh status but found other error said "/etc/apps/TA-eStreamer/bin/encore/" does not exist

After searching for any info, we thought that there is problem with the pkc12 file, so we generated new file from FMC and put the new one into /TA-eStreamer/bin/encore and put the password from setup Page. Unfortunately while we try to save this configuration, another error occurred that said Encountered the following error while trying to update: Splunkd daemon is not responding: (u"Error connecting to /servicesNS/nobody/TA-eStreamer/apps/local/TA-eStreamer/setup: ('The read operation timed out',)",)

Any idea how to fix this?

Thank you for all your assistance.

[vinz2020]

same issue for me with the last app release 3.6.8
https://splunkbase.splunk.com/app/3662/

and FMC v6.4.0.7

I received some data events during a few minutes and then the estreamer process goes down (and looping for a while)

[mkemp]

Per the deployment guide we have three options: https://www.cisco.com/c/en/us/td/docs/security/firepower/630/api/eStreamer_enCore/eStreamereNcoreSpl...

■        0: Send all events from the earliest point available on the Firepower Management Center

■        1: Send all events that occur after receiving the client request

■        2: Use a bookmark to pick up where we left off. First run is from 0

So, first modify the file to use option 0. Restart the encore and leave it running some time and verify if you see events.

After that you can modify the file to option 1 and restart the encore again and verify if events are seen in encore.

[jbrinkman]

I would take a look within $SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore. Should have a file referencing the FMC you had configured to reach out to. Mine was IP-port_proc.pid. Rename or delete that file. Restart Splunk. I began ingesting immediately following the restart.

[shahrukhvp]

This resolved my issue. 

[Mesa_Splunkr]

Anything recent on this issue? I just discovered I am having the same. Just digging in now. Thanks for posting and your comments.

[mhessel]

I had to back out to the older version 3.0 and the events came back. I don't know why, but the estreamer process will start, and I get a short burst of events, but stops and nothing happens from that point on.

[xtrjx]

I am experiencing the same issue. The 3.5 version will retrieve a burst of events but then stop. Every time I restart the estreamer processes, either from the Splunk UI by enabling/disabling eNcore, or from the shell using kill -HUP on the estreamer processes, or by restarting splunk entirely, I always get a burst of events when eNcore starts but then it will stop. I can wait hours and still not get a new event until I restart. Then I will get a burst of events from the last "bookmark". I've tried changing many of the values in estreamer.conf including connectTimeout, responseTimeout, workerProcesses, with no luck. I am running on a server with 16 CPUs and 64GB RAM so have plenty of power. estreamer events get processed just fine with the old 2.2.1 version of the estreamer app/addon.

[mhessel]

This might be related to a bug on the FMC, where the estreamer service will peg a cpu at 100 percent if FireAMP or File events are enabled. Cisco tracks this as bug #CSCvj07843

I was able to get this working by disabling the event types in the FMC, only allowing intrusion and malware events, and now I am getting events in splunk, but that is still running 30 minutes to an hour behind. This does not seem to be related to the estreamer usage on the splunk/client side, but on the FMC instead.

I'm guessing with the additional protocol support added in 3.5.0, the estreamer service on the server side is slowing down too much.

From the bug report, fixed versions of the FMC are 6.2.2.4, and 6.2.3.2.

[xtrjx]

Additional details: I enabled DEBUG estreamer logging. Now I noticed the following log messages estreamer.log that correspond with the data flow stopping:

2018-07-28 06:07:29,722 Decorator    DEBUG    Stashing sequence 80874; buffer: 1

Those messages will repeat with the stashing sequence incrementing by about 100 at a time and buffer number incrementing from 1 to n until I restart the estreamer processes. For example, this is the last log entry after letting estreamer run all weekend:

2018-07-30 15:36:41,905 Decorator    DEBUG    Stashing sequence 167072; buffer: 857

[Blackburn2413]

Same boat here. Same message runs all day now. estreamer shows as running in the Cisco Security plugin, but no logs, alerts, results... Nothing.