Estreamer failing after 6.2.01 upgrade

[panovattack]

Recently updated FMC to 6.2.0.1. Estreamer client now only sends 5 or so events and then the estreamer client fails, both on Splunk and host-based client testing. Also, the server does not seem to respond to changes in the event type delivery options. Is the estreamer APP not compatible with FMC 6.2.0.1?

[douglashurd · Accepted solution]

A new Splunk Firepower solution is now available if you are using Firepower version 6.x. You can download the new eStreamer eNcore for Splunk and the separately installable dashboard from the two links below:

eStreamer eNcore
https://splunkbase.splunk.com/app/3662/

eNcore Dashboard
https://splunkbase.splunk.com/app/3663/

It is free to use and well documented but if you would like to purchase a TAC Support service so that you can obtain installation and configuration assistance and troubleshooting you can order the software from Cisco (support obligatory with this purchase). The Product Identifier is: FP-SPLUNK-SW-K9.

Regardless of whether you take up the support option or not, updated versions will be made available to all free of charge and posted on Splunkbase as well as Cisco Downloads.

[panovattack]

eNcore + 6.2.2 solved our issue. However we still struggle with finding the multi-processor settings for eNcore. eNcore runs as a single thread, not sure how to make it multi processor. we configured our HF just to do this.

[BHeindel]

There is a hotfix available for FMC code 6.2.0.1 that we applied, that makes streamer function MUCH better.

I had to open a TAC case to get the hotfix for 6.2.0.1 - (bug CSCve44987)

I am told the same hotfix is rolled up into update 6.2.0.2 if you prefer to go that route.

[ChrisBell04]

@BHeindel@idahopower.com
confirmed that 6.2.0.2 (build 51) fixes a lot of the estreamer issues present in 6.2.0.1.

[jkleensang]

I have the same issue, tho I can't confirm it was due to an upgrade... still waiting for word back from that team. Not sure why, but when I start the eStreamer client manually, collection seems to continue past the initial lump of events (behaves normally).
For reference, I started it with:

/usr/bin/perl /opt/splunk/etc/apps/eStreamer/bin/estreamer_client.pl -d -c /opt/splunk/etc/apps/eStreamer/local/estreamer.conf -l /opt/splunk/etc/apps/eStreamer/log/estreamer.log

[jkleensang]

shucks Looks like that solution has a limited lifetime... worked for a couple hours and then died again.

[jkleensang]

Seems that the root of our problem was that /var filled on the FMC - once that was clear and all services restarted, eStreamer events started flowing again.

[dw385]

How did you clear /var or what is preventing it from happening again?
I don't have direct access to the FMC but I'm told this seems to be our issue too. Trying to work with Cisco for resolution. At the moment they want to delete our estreamer connection and recreate it. I'm not convinced but hope to have this done in the next day or so.

[jkleensang]

Honestly, I don't know. This device is managed by another team so I don't have any access to it either. It looks like it's just a linux based system, so deleting anything should be pretty straight forward 🙂 There were many mentions/concerns in the release notes for the 6.2.x versions about space, and I think I saw some recommended actions too, so maybe start there if Cisco support doesn't pull through for you.

[dw385]

Also seem to be experiencing the same issue, likely after the 6.2.0.1 upgrade. The debug log from the Splunk estreamer client seems to suggest it connects OK and is "waiting for more logs". I'm told on the FMC there are errors "May 4 18:38:58 FS-95 SF-IMS[18510]: [18510] EventStreamer child(SPLUNK-IP):ConnectionHandler [ERROR] Error sending a message: Not connected"

Although we didn't notice it immediately I do believe it stopped working with 6.2.0.1 upgrade.

Restarting Splunk Estreamer, toggling options doesn't seem to get me more than a few events.
We're configured to log everything but packets.

[BHeindel]

I am also experiencing this issue, log files are coming in from Sourcefire and piling up on the Splunk server, just not getting ingested

[panovattack]

when running the search:

search index=_internal "/etc/apps/eStreamer/log" sourcetype!=splunkd_remote_searches

we see the following:

INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/etc/apps/eStreamer/log/estreamer.log.1494267835

do you see the same?

[panovattack]

After a reboot of the FMC, the reference client (latest version) grabs events correctly, however, the estreamer splunk app client still fails after 5 or so events, and only discovery events.